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ABSTRACT 

The develop^^ent and validation o£ a Portuguese oral 
language test are described. The test consisted of five item types: 
personal conversation, giving directions, description of picture 
sequences, topical discourse, and oral task completion based on 
printed instructions. Three preliminary forms of the test were 
administered to a group of language students and native speakers, and 
follow-up questionnaires elicited information about format and item 
preferences. Based on responses, minor changes were made in test 
items, timing, and instructions. Two versions of the test's taped 
portion, one in Brazilian and one in Lusitanian Portuguese, were 
made. A validation study assessed interrater reliability, 
interchangeability of the three test versions, and the comparability 
of responses to the taped interview versus a live interview. 
Interrater reliabilities were found to be uniform across the test 
forms and live interview, and interrater reliability was not 
adversely affected by the semi*direct format. Feedback on test 
construction and difficulty was generally supportive, althougn most 
respondents preferred the live interview to the taped version. The 
test has been operationalized and is currently available. Development 
of similar tests for Hausa, Hebrew, and Indonesian will follow. Nine 
figures are provided; test forms and test administration information, 
comprising two-thirds of the document, are appended. (MSE) 
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OySRVIBW OF TEE PROJECT 

During the past decade, thanks primarily to the efforts of 
the American Council on the Teaching of Foreign Languages (ACTFL) 
and the Educational Testing Service (E7S) , and with the assistance 
of several government language training agencies under the auspices 
of the Interagency Language RoundtaUble (ILR) , a major theoretical 
and practical development in the field of foreign language 
asses?5inent has taken place. This development is the application 
of a "proficiency" orientation in the testing of foreign language 
competence. This movement has also had an effect in the adoption 
of a proficiency approach to language instruction in many qpiarters. 
Its impact is even now being felt, for example, in Title VI of the 
Higher Education Act, Section 602, "Graduate and Undergraduate 
Language and Area Centers." This section states that recipients 
of federal stipends "shall be individuals who are engaged in a 
program of competency-based language training, or in a program 
developing competency-based language training...." According to 
The Federal Register (vol. 52, no. 19, p. 37069, paragraph 
657.5(f)), competency-based language training means "a training 
program which had defined functional language use objectives and 
whose evaluation measures for students can be linked to established 
national standards." 

Leading the way in foreign language proficiency measures is 
the oral proficiency interview, a direct face-to-face evaluation 
of the foreign language learner's competence conducted by trained 



interviewers and raters. However, for the less coniaonly taught 
languages (LCTLs) , the practical problems of economics and 
organization impede having adequate numbers of individuals 
available to test competency via the live interview. Semi«-direct 
testing (using recorded and printed stimuli and recorded responses) 
is the most efficient and feasible approach to proficiency 
measurement in the IXTTLs. This approach eliminates the need to try 
to sustain a costly and labor intensive face-to-face (direct) oral 
proficiency interview program for low-volume languages whose 
enrollment figures may be unstable from year to year. However, it 
ensures the benefits derived from a continual assessment program 
as the impetus for competency-based learning for students of the 
LCTLs. 

Funding for the current project covers the development of four 
semi-direct tests under the guidelines established in the proto- 
typical development project of a semi-direct test for Chinese 
(Clark, 1986; Clark and Li, 1986). In year one of the project, 
which this report covers, a semi-direct test was developed, 
validated and operational ized for Portuguese. In year two, tests 
will be developed for Hebrew, Hausa and Indonesian. 

MMOR PROJECT ACTIVITIB8 

The day-to-day work of the project was conducted at the Center 
for Applied Linguistics (CAL) in Washington, D.C. Charles W. 
Stansfield served as Project Director and Dorry Kenyon as Test 
Development Specialist. A Test Development Committee was formed 



which included, in addition to the above, three experienced 

instructors of Portuguese with training in using the ILR and/or 

ACTFL proficiency rating scales: Dr. Ricardo Pciva (Georgetown 

University), Ms. Fatima Doyle (Foreign Service Institute Language 

School) and Ms. Ines Ulsh (United States Govemoent) • Mj. Gayle 

Belcher completed the team as the artist for the test. 

The committee met on a regular basis from January to March, 

1988, to develop the specific items for the test. These items were 

based on the question types used in the semi-direct test of 

Chinese. Following is a description of each item type (sample 

items may be found in the Examinee Handbook . Appendix I) : 

!• Personal Conversation, 

This section corresponds to the "warm-up" section of the 
direct interview, in this section, the examinee listens to 
conversational questions about his/her family, education, 
hobbies, etc. in Portuguese and responds to each question as 
it is asked. There are 12 to 13 such questions on each form. 
This is the only section in which Portuguese is used on the 
tape. Nhen ordering the PST, a Brazilian or a Lusitanian 
version may be requested. 

For each of the following question types, the examinee is 
given between 15 and 30 seconds to prepare an answer before being 
required to speak. Time for giving an answer ranges from 45 to 1 
minute and 45 seconds. 

Giving Directions. 

The examinee is shown a pictorial map in the test booklet and 
is instructed to give directions between two points on the 
map. The question is contextualized in that examinees are 
instructed that they are giving this information to a friend 
over the phone. 



3. 



Detailed Descrintion. 



The examinee is shown a drawing in the test booklet and is 
instructed to describe the picture in as Buch detail as 
possible. Each picture contains not only a variety of objects 
but also of actions. This question is also contextual i zed so 
that the examinee knows the specific audience being addressed 
and the purpose of the description. 

4. Picture Sequences. 

The examinee is instructed to speak in a narrative fashioi^ 
about a sequence of four or five pictures shown in the test 
booklet. There are three questions of this type; one each for 
past, present and future time narration. Again all questions 
are contextual ized so that the examinee is given a specific 
audience and a specific reason for the narration. 

b. Topical niscoursc. 

The examinee is instructed to talk about selected topics 
involving different discourse strategies. These strategies 
include explaining a process, supporting an opinion and 
talking about a hypothetical situation. There are five such 
topics, each printed in the test booklet. 

6. Situations. 

The examinee reads a printed description of a real-life 
situation in which a specified audience and communicative task 
are identified. The examinee is then instructed to carry out 
the specified task. The tasks range from making simple 
requests to giving a formal toast. 



TRIALIMO or THE TB8T FOSIIS 

Three preliminary forms of the PST, each containing a few more 
items than would ultimately be required, were developed. These 
three forms were trialed on 15 learners of Portuguese at Georgetown 
University (l form each) as well as on 2 native speakers of 
Portuguese (1 Brazilian, 1 Lusitanian) in order to ensure that the 
questions were clear, understandable and working as intended and 
to check the appropriateness of the pause times allotted on the 



tape for examinee responses. Assisting with the logistics of the 
trailing at Georgetown Uni"ersity were Dr. Ricardo Paiva of the 
Portuguese Department and Ms. Jackie Tanner, director of the 
Language Media Center. Nine of the learners took the test in a 
language lab setting; the others were administered the test on an 
individual basis using two tape recorders. in all cases, upon 
completion of the test, the examinees responded to a detailed 
questionnaire about it (Appendix C) . When possible, they were also 
questioned aboi?t the test in person. The student responses on tape 
were listened to by a Portuguese speaking member of the test 
committee, who also responded to a questionnaire about the test 
(^pendix D) . Whenever the test was individually administered, a 
member of the test development committee observed the examinee take 
the test and recorded notes on the same form. Ms. blsh, on the 
basis of her experience as an ACTFL rater, assigned a rating to all 
tapes and completed a third questionnaire for each examinee 
(Appendix E) . 

The student questionnaires end comments were summarized and 
the committee met to discuss revisions. Minor modifications were 
necessary in the questions; in most cases this involved 
clarification of ambiguous items in the pictures. The original 
pause times were generally confirmed, though they were reduced in 
the opening conversation section and lengthened or shortened in 
some of the topics and situations. More time was given to prepare 
and answer the question involving supported opinion. 

In response to student comments, instructions were added to 
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the general directions explaining that, although the location of 
questions varied between Brazil and Portugal, the examinee would 
only need to answer in the type of Portuguese he/she normally uses. 
In addition, it was decided to provide two different versions of 
the tape (one for Brazilian and one for Lusitanian Portuguese) in 
the warm-up conversation section. This decision did not alter the 
wording of the scripts significantly and test booklets remained 
the same for both versions. Also based on student comments, items 
were chosen for deletion from the final forms. 

The final printed forms of the test booklets appear in 
Appendix A of this report. The final script is in Appendix B. 
(Note that as the PST is an operational and secure test, these two 
appendices are not available outside of the Department of 
Education. For further information on the tests, contact the 
Coordinator, Portuguese Speaking Test, Center for Applied 
Linguistics, iiis 22nd Street, NW, Washington, DC 20037.) 

VlkLIDATIOM STUDY 

A research study was designed and carried out to validate each 
of the three forms of the test. The study sought to answer the 
following questions: 

1. Can this test, which involves spoken responses in 
Portuguese, be scored reliably by different raters? 

2 . Are the three separate forms of the PST interchangeable, 
i.e., do they produce similar examinee results 
independently of the particular form administered? 

3. Do the recorded responses produce the same score as a 
regular live interview for any given examinee? 



To answer these questions, a research design was prepared 
involving 30 subjects. Each subject was administered two forms of 
the PST and the oral interview. The design controlled for the 
order of administration, with half of the subjects receiving the 
oral interview first and half the taped test first. The third test 
for all subjects was a taped test. The design also controlled for 
the order of the PST tests in that 10 subjects took Forms A and B» 
9 subjects took A and C, and 11 subjects took B and C, and within 
each group half of the subjects took one form first and then the 
other. The design also attempted to control for proficiency level. 
Based on teachers' observations, siibjects were assigned into a 
"high" group and a "low" group. Half of each of the A-B,A-C,B-C 
groups were initially "high", half initially "low." 

The subjects were predominantly undergraduate students who 
had completed a year or more of Portuguese study at the University 
of Maryland, College Park, and current and former students in the 
Portuguese Department at Georgetown University. One subject was 
a CAL employee and another worked at the Foreign Service Institute. 
Each subject received an honorarium of $20.00 for participating in 
the study. Assisting with the logistics of the study at the 
University of Maryland were Dr. Regina Igel and Mr. Manual Gordo 
of the Portuguese Department and Dr. Cathy James, assistant 
director of the Language Media Center, and at Georgetown 
University, Dr. Ricardo Paiva of the Portuguese Department and Ms. 
Jackie Tanner, director of the Language Media Center. 



Two raters were used in the design: Fatina Doyle (Rater 1), 
FSI certified rater from the Foreign Service Institute and Maria 
Antonia Cowles (Rater 2) , ACTFL certified rater from the University 
of Pennsylvania. Ms. Doyle conducted 16 of the face-to-face 
interviews and Ms. cowles conducted 14. 

The majority of the subjects were tested at their respective 
university in one session between 2 and 1/4 and 3 hours in 
duration, except in a few cases where conflicts in the subject's 
schedule nade this impossible. In such instances, testing was done 
on separate days. 

The live interview was tape recorded. Upon completing the 
live interview, the interviewer assigned a rating to the subject. 
After all interviews were completed, the recorded tape of the live 
interview was given a second, independent rating by the other 
rater. Each rater also rated each of the recorded PST test tapes, 
anonymously and in random order. After all the ratings were 
completed, subjects were sent all their test results in the nail: 
the scores of the two raters on the live interview and on each of 
the PST forms. 

Rating of both the live interview and the tape-based semi- 
direct tests was done on a scale combining both ACTFL and ILR 
rating scales with weights assigned as follows: 
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ACTFL/ILR Level 



Coded as; 



Novice-Low 

Novice-Mid 

Novice-High 

Intermediate-low 

Intermediate-Mid 

Intermediate-High 

Advanced 

Advanced-Plus 

Superior/Level 3 

Level 3-t- 

Level 4 

Level 4-t- 

Level 5 



0.2 
0.5 
0.8 
1.0 
1.5 
1.8 
2.0 
2.8 
3.0 
3.8 
4.0 
4.8 
5.0 



The system of score coding 9bove is based on the ILR 0 to 5 
rating scale and is intended to assign an appropriate nvmerical 
value to the proficiency level descriptions. For example 
proficiency at an Advanced-Plus level is characterized by many of 
the same features as at the Superior/3 level, though the examinee 
cannot sustain the performance. Thus, the numerical interpretation 
falls closer to 3.0 than mid-way between the two, as may be 
expected . 

The several tables below provide descriptive statistics, 
interrater reliabilities and parallel-form reliability data 
obtained in the study. Table l shows the mean score, standard 
deviation and other basic statistics for the ratings assigned by 
each of the two rater to subject performances on each of tho semi- 
direct test forms and on the live interview. 
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Table 1 

Descriptive Statistics for Scorinc, Levels Assigned, Tape and Live 
Tests 

Minimum Maximtun Standard 
Test Form Score Score Mean Deviation 



A (N=19) 



Rater 1 
Rater 2 



B (N=21) 



Rater 1 
Rater 2 



C (N=20) 



Rater 1 
Rater 2 



0.2 
0.2 



0.2 
0.2 



0.2 
0.2 



All Foms (N=60) 



Rater 1 
Rater 2 



0.2 
0.2 



Interview (N-30) 



Rater 1 
Rater 2 



0.2 
0.2 



4.0 
3.8 



4.8 
4.0 



4.8 
4.0 



4.8 
4.0 



4.8 
4.8 



1.74* 
2.01 



1.90 
2.04 



1.75 
1.83 



1.80 
1.96 



1.92 
1.96 



1.02 
0.94 



1.17 
0.94 



1.14 
1.03 



1.10 
0.96 



1.19 
1.13 



* The difference in these means was significant at the p>.05 
level . 



Table 2 shows the frequency of ratings given by the two raters 
on the live interviews (N=30) and on all forms (N»60) . 
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Table 2 
Frequency Distributions 

Live Interview Ratings (Rater 1) 

Cumulative Cumulative 



Rat.ing 


Frequency 


Percent 


Frecfiiencv 




0.2 


2 


6.7 


2 


6.7 


0.5 


3 


10.0 


5 


16.7 


0.8 


1 


3.3 


6 


20.0 


1 


4 


13.3 


10 


33.3 


1.5 


2 


6.7 


12 


40.0 


1.8 


4 


13.3 


16 


53.3 


2 


5 


16.7 


21 


70.0 


2.8 


4 


13.3 


25 


83.3 


3 


1 


3.3 


26 


86.7 


3.8 


2 


6.7 


28 


93.3 


4 


1 


3.3 


29 


96.7 


4.8 


1 


3.3 


30 


100.0 



Live Interview Ratings (Rater 2) 

Cumulative Cumulative 



Rating 


Frequ »ncy 


Percent 


Frequency 


Percent 


0.2 


1 


3.3 


1 


3.3 


0.5 


3 


10.0 


4 


13.3 


0.8 


2 


6.7 


6 


20.0 


1 


3 


10.0 


9 


30.0 


1.5 


3 


10.0 


12 


40.0 


1.8 


3 


10.0 


15 


50.0 


2 


5 


16.7 


20 


66 7 


2.8 


5 


16.7 


25 


83.3 


3 


2 


6.7 


27 


90.0 


3.8 


1 


3.3 


28 


93.2 


4 


1 


3.3 


29 


96.7 


4.8 


1 


3.3 


30 


100.0 
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Rating of PST Test Forms (Rater 1) 



Cumulative Cumulative 



ng 


Frequency 


Percent 


Frequency 


Percent 


0.2 


5 


8.3 


5 


8.3 


0.5 


5 


8.3 


10 


16.7 


0.8 


4 


6.7 


14 


23.3 


1 


4 


6.7 


18 


30.0 


1.5 


10 


16.7 


28 


46.7 


1.8 


12 


20.0 


40 


66.7 


2 


4 


6.7 


44 


73.3 


2.8 


5 


8.3 


49 


81.7 


3 


7 


11.7 


56 


93.3 


4 


2 


3.3 


58 


96.7 


4.8 


2 


3.3 


60 


100.0 



Rating of PST Test Forms (Rater 2) 



Cumulative cumulative 



ng 


Frequency 


Percent 


Frequency 


Percent 


0.2 


5 


8.3 


5 


8.3 


0.5 


1 


1.7 


6 


10.0 


0.8 


3 


5.0 


9 


15.0 


1 


2 


3.3 


11 


18.3 


1.5 


6 


10.0 


17 


28.3 


1.8 


16 


26.7 


33 


55.0 


2 


11 


18.3 


44 


73.3 


2.8 


3 


5.0 


47 


78.3 


3 


8 


13.3 


55 


91.7 


3.8 


3 


5.0 


58 


96.7 


4 


2 


3.3 


60 


100.0 



These statistics indicate that each form of the test was taken 
by 3 firoup of examinees that varied widely in oral language 
proficiency. similarly, the ratings assigned by both raters 
reflected this range. The mean scores for each rater were very 
similar, indicating that the raters were almost equal in their 
degree of severity. However, Rater 1 was consistently more severe 
than Rater 2, although this difference was minuscule except on form 
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A, where thr difference in average rating was a very acceptable ,26 
of a level on the ILR scale. Using a t-test for the difference 
between paired means, only that between ratings on form A was 
statistically significant (p>.05). 

The degree of agreement between the absolute ratings may be 
seen from the following two cross-tab diagrams. First, Table 3 
presents the ratings of Rater 1 (down) against the ratings of Rater 
2 (across) for the Live Interview. 



Table 3 

Crotttabulations of Live Interview MttnQt (N«30) 



later 1 Cdowi)/ / later 2 Cacroas) 



rrequencyj 0.21 




0.8| 










2.8| 




3.8| 


4| 4.8 


0.2 1 11 


.. M 


.. 1 


















?:M 1.. 


.. M 


...M 


















0.8 1 1 


1 










1 










M 1.. 


... 1 




















1.. 


1 


... 1 


















!:?.! 1.. 


1. 


.. 1 


















?.l 1.. 


1 




















?:?.! 1.. 


... .j 




















M 1.. 


1 




















L 


1 . 




















M L 


1 


... 1 


















J.. 


1 





















Total 



2 
3 
1 
4 
2 
4 
5 
4 
1 
2 
1 
1 



Total 



30 



For the Live Interview, there was total agreement in 63% of 
the ratings. m only three cases of the 30 interviews was the 
disagreement in the rating more than one step of the rating scale 
away. There was only one case of a complete level disagreement, 
where Rater 1 gave 3.8 and Rater 2 gave 2.8. 
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Table 4 presents the ratings of Rater 1 (down) against Rater 
2 (across) for the PST forms. 



Table 4 

CrosttJibulatiom of PST form Ratings (60 ratings) 
Rater 1 (down) / / Rater 2 (across) 



rre^iency| 0.2| 0.5| O.Sj 1| 1.51 1.81 2 1 2.8 1 3l 3.8] 4] Total 



0.5 1 1 ..M ^ I M 1 1 1 1 1 1 1 5 

1 1 M ^ 1 1 1 1 1 i ^ 

i^iiiij^r.^j**!™!!^^ ^ J I I I I ^ 

] I I I 1 1 * Ti T\ 1] I I I I 10 

1.8 I I I I 1 1 61 6 ] ] ] I I 12 

] I ] I j ] ] 3] 2] j j 5 

] ] I I I j j j j I 2 1 2 

Total 5 1 3 2 6 16 11 3 8 3 2 60 



From Table 4 ve see that the agreement of the absolute ratings 
was again quite high. There was total agreement in 48% of the 60 
PST form ratings. For only 13% of the ratings was the disagreement 
more than one step avay on the rating scale. There were only two 
cases of a complete level disagr^eement : in one instance Rater 1 
gave 0.8 where Hater 2 gave 1.8 and in another Rater 1 gave a 1 
where Rater 2 gave a 2. 

The two tables above show that Rater 2 tended to be slightly 
more generous than Rater 1, especially on the taped forms in the 
fine differences between Intermediate-Mid (1.5), Intermediate-Hiqh 
(1.8) and Advanced (2.0). 
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Interrater reliabilities (Pearson product -moment correlations) 
between the ratings assigned by Rater 1 and those assigned by Rater 
2 for each of the semi-direct test forms and for the live interview 
are shown in Table 5 below* 



Table 5 
Interrater Reliabilities 

Test Form Correlation 

A (N=19) .93 

B (N=21) .98 

C (N=20) .96 

All 60 Forms Paired .95 

Inte'^iew (N=30) .94 



These interrater reliabilities are all uniformly high across 
the three test forms and the live interview. Interrater 
reliability was not adversely affec;:ed by the semi-direct test 
format. This suggests that the PST elicits a sample of speech as 
ratable as the live interview. 

On performance-based tests such as the PST, there is an 
increased concern for test-retest reliability. This form of 
reliability measures the degree of inconsistency in examinee 
performance on two separate administrations of the same tes-t. The 
amount of inconsistency reflects the degree to which the test score 
may be confounded by such inconsistency. Therefore, it is 
important to examine this factor. However, on a test with a 
limited number of questions such as the PST. it is not wise to 
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administer the same test twice, since the first sitting will serve 
to instruct the examinee in the task at hand, (For a thorough 
discussion of this ••reactivity effect, see Stansfield and Ross, 
In press.) Under such circumstances, it is preferable to 
administer different forms of the test while still using the sane 
rater to score the performance. This type of reliability is known 
as parallel-form reliability, which is the degree of correlation 
between scores on two forms of the test. 

Parallel-form reliabilities for the same subject taking two 
different test forms, with the same rater scoring both forms, are 
shown in Table 6. (Note that in this and subsequent tables, the 
three different PST forms are combined for analysis in a logical 
fashion by using the rating an examinee was awarded on the first 
form he or she was admin 'Litered with the rating awarded on the 
second form he or she was administered.) 



Table 6 

Parallel-Form Reliabilities (Same Rater) 

Tests Taken By Subject Rater 1 Rater 2 

Forms A and B (N=10) .99 .97 

Forms A and C (N=9) .99 .93 

Forms B and C (N=ll) .99 .97 

All Forms (N=30) .99 .96 

(First Form/Second Form) 



The statistics indicate that the parallel form reliability of 
the PST is very high. With the first rater, the parallel-form 
reliability was nearly perfect .99. With a different rater. Rater 
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2, the parallel-form reliability was also very high .94. Such 
favorable statistics provide strong support for the proposition 
that each form of the PST elicits a sample of speech that is 
uniformly challenging to the examinee. The fact that the parallel- 
form reliability was high for two different raters supports the 
claim that the sample of speech elicited by different forms is 
equally ratable. 

In summary, the evidence from Table 6 warrants the conclusion 
that natural variations in examinee oral language performance are 
adequately controlled for by the PST format. 

Table 7 shows parallel-form reliabilities for subjects taking 
two different test forms, with each form scored by a different 
rater. 



Table 7 

Parallel Form Reliabilities (Different Forms and Raters) 



Rater/Form Combination Correlation 

Rater 1/Form A - Rater 2/Form B (N=10) .98 
Rater 1/Form A - Rater 2/Form C (N=9) .94 
Rater 1/Form B - Rater 2/Form A (N=10) .92 
Rater 1/Form B - Rater 2/Form C (N-11) .97 
Rater 1/Form c - Rater 2/Form A (N=9) .97 
Rater 1/Form c - Rater 2/Form B (N=ll) .97 
Rater l/First Forms- 
Rater 2/Second Forms (N=30) .95 
Rater l/Second Forms- 
Rater 2/First Forms (N=30) .95 



This type of parallel-form reliability involves error that 
can be attributed to natural variation in examinee speech, error 
that can be attributed to differences in test form, and error that 
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can be attributed to differences in raters. Thus, it may be viewed 
as a lower-bound estimate of the reliability of a PST score. Again 
the reliabilities here are all high, and the average reliability 
under these severe conditions (different forms and different 
raters) is .95. 

Correlations of semi-direct test scores with the live face- 
to-face interview are given in Table 8 below. These correlations 
are evidence of the validity of the PST as a surrogate live 
interview. (Note that in pairing the two ratings, each examinee's 
interview rating was used twice to correspond to each of that 
examinee's t%*o PST form ratings.) 



Table 8 

Correlations with Live Interview 
Rater/Form Rater 1/Interview Rater 2/Interview 

Rater l/Form A (N=19) .96 .91 

Rater l/Form B (N»21) .95 .94 

Rater l/Form C (N=20) .93 .92 

Rater 1/All Forms (60 pairs) .95 .93 

Rater 2/Form A (N»19) .92 .90 

Rater 2/Form B (N=21) .91 .91 

Rater 2/Form C (N=20) .93 .92 

Rater 2/All Forms (60 pairs) .92 .91 

All Matched Interviews/ Forms .93 
(120 pairs) 



Again, the correlations are all high. The average correlation 
based on 120 pairs of ratings (30 subjects x 2 PST forms x 2 
ratings, correlated with the score c;ssigned for the live interview) 
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was .93. Such results support the claim that the PST is a valid 
measure of oral language proficiency that can be substituted for 
a live interview. 

The degree of agreement in absolute ratings given on the Live 
Interview with ratings given on the same examinee's PST may be seen 
from the following cross-tab diagrams. In Table 9 all 120 pairs 
of interview ratings (down) with PST ratings (across) are 
presented. 



Tablt 9 
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From the table we see that in 41% of the capes there was an 
absolute agreement between the two ratings, while in an additional 
50% of the ratings the difference was only one away step on the 
rating scale. Thus, for 91% of the pairs, the rating on the live 
interview and the rating on the PST was equal to or l«iss than one 
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step on the rating scale. in only four pairs was there 
disagreement of the magnitude of a complete level; in three cases 
an interview score of 2.8 was given while a PST score of 1.8 was 
given, and in one case a rating of l was given on the interview 
while a rating of 2 was given on the PST. Thus, besides the high 
correlations documented above, the absolute values given to 
examinees on both the Live Interview and the PST were extremely 
close. 

As a general summary of the statistical information above, it 
■ay be stated that all three forms of the experimental semi-direct 
test reveal high interrater reliabilities, with Pearson product- 
moment correlations ranging from .93 to .98. Parallel form 
reliabilities are also very high, even under the most "severe" 
conditions (i.e., different raters rating two different forms), 
where correlations range from .92 to .98; with the same rater, 
correlations range from .93 to .99. The correlations with the live 
interview are also very high; with the same rater they range from 
.90 to .96 and with different raters from .91 to .94. 

SUBJECT RESPONSE TO THE TEST 

Feedback information from the subjects who participated in 
the study on various aspects of their experience with and opinions 
about both types of testing procedures were elicited by means of 
a short questionnaire (Appendix F) . The questionnaire was given to 
the subjects directly after completing the last of the three tests 
and completea and returned, in most cases, before leaving the 



testing room. In a few cases the questionnaires were mailed back 
at a later date. All 30 subjects conpleted the questionnaire for 
a 100% participation rate. 

The answers to the examine questionnaires are given in graphic 
summary form below. Written comments in response to the 
questionnaire are presented in Appendix G. 

The first two questions sought to elicit from the subjects 
the extent to which they felt their Portuguese speaking ability 
had been probed by the two types of test: the live interview and 
the PST. 

(1) Over the course of the live iBtervie y . do you f««l your 
■aziauB level of speaking ability in Portuguese was adequately 
probed by the tester? 

(2) over the course of the taped teat , do you feel that the 
descriptions, narratives, situations, and other types of questions 
in the test were adequate to probe your HaxiHum level of speaking 
ability in Portuguese? 

Figures l and 2 reveal that exactly same high percentage of 
students responded positively to both questions. This suggests 
that for the most part the subjects felt their speaking ability was 
being adequately tested and they found no difference in the ability 
of the two test formats to test the depth and thoroughness of their 
present Portuguese speaking ability. 
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The next two questions focused on whether the subjects 
perceived any unfair questions on either test format. 

(3) In the live intarvieif. were there any questions asked or 
speaking situations required which you felt were in any wav 
•unfair*? ' 

(4) In the taped teats, were there any picture/descriptions, 
narratives, situations, or other questions that you felt were in 
any way * unfair*? 

As shown in figure 3, all subjects replied that they felt 
there were no unfair questions in the live interview, although one 
subject narked both categories. For the taped test, as shown in 
figure 4, only 5 subjects felt there were unfair questions on the 
PST. This nay be due to the fact that some of the subjects tested 
were beneath the recommended ability level of the test (i.e., they 
were discovered to be novices). In such a case, the live 
interviewer can adapt the interview or cut it short, whereas in the 
taped test, the examinee is asked every question. in any case, 
only a very low percentage of subjects felt there were 'unfair' 
questions on the taped test. 



23 



Unfair Questions in Live Interview? 



120 
100 

ao 

60 
40 
20 
0 



100 














1 




0 



Mo 



Figure 3 



Unfair Questions in Taped Test? 
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The next two questions focused on the subject's affective 
perceptions of the test. 

(5) In which of the two types of tests (live interview or taped 
test) did you feel more anxious or nervous? 

(<) Which of the two types of tests (live interview or taped test) 
did you feel was more difficult? 

Because the semi-direct mode of testing may be unfamiliar and 
perhaps 'unnatural* to students in general, it would not be unusual 
for a large percentage of the students in this study to feel more 
nervous in the taped test than in the live interview. However, 
less than 70% of the subjects felt more nervous in that mode, while 
over 30% of the subjects felt more nervous or the same amount of 
nervousness in the live interview. 

Question 6 focused on perceived difficulty. Despite the fact 
that subjects did approximately the same on both tests (see 
correlations above) , the vast majority perceived the taned test as 
more difficult. Perhaps some of the individual comments are 
enlightening (see Appendix G) ; these seem to revolve around 
the timed pauses, the length of the test (for subjects who were 
not at the recommended level) , and discomfort in talking tr a 
machine, it appears the 'unnatural* format contributed heavily to 
perceived difficulty. 
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Questions 7 and 8 focused on technical qualities of the taped 

test. 

(7) in ttaa taped test, were the pauses for your responses usually 
long enough for you to respond as fully as you wished or were able? 

(8) Where the directions on the taped test clear? 

The majority of the subjects had no problem with the timed 
pauses in general. From the individual comments (Appendix G) and 
the fact that 3 subjects marked more than one category in response 
to this (question , pause time was in general an individual concern , 
varying from person to person and from question to question. Given 
the individual variation mentioned above, in a taped test with 
timed pauses such as the PST, the fact that 70 percent of the 
subjects felt the pauses were about right is significant. 

It is also significant that 100% of the subjects felt the PST 
directions were clear. This is gratifying as there is no 
possibility in the taped-test mode for examinees to ask questions 
if something is misunderstood once Part One of the test is begun. 
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Taped Test Directions Clear? 
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Question 9 is the •catch-all • summary question. 

(9) Which of the two types of tests did you prefer— the live 
interview or the taped test? 

The majority choose the live interview. From the comments in 

Appendix G, we can see that this is probably a r<if lection on the 

live interview testing mode, which seemed more natural, rather than 

a reflection on the technical quality of the taped test. However, 

14% of the subjects either preferred the taped test or had no 

preference, and two comments suggest that the taped test was more 

interesting and challenging than the live interview. 
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In summary, it appears that though the subjects were very 
positive about the content, technical quality and ability of the 
taped test to probe their speaking ability, the unfamiliar mode of 
testing and perceived *unnaturalness* of speaking to a machine 
caused a greater perceived difficulty and more nervousness than the 
live interview. Thus, the majority of the subjects said they 
preferred the live interview to the taped test. Nevertheless, 
given the extremely high correlations between the two types of 
tests and the positive response to the taped test quality it 
appears that the taped test may confidently be used as an 
alternative, albeit "second choice" in the examinee's eyes, to oral 
proficiency testing via the live interview. Moreover, it is 
expected that e>^minees who are more prepared for the test through 
the Examinee Handbook (see next section) may find the testing mode 
less threatening than the subjects participating in the validation 
Btudy who went to both the live interview and the taped tests 
without any special advance preparation in order to avoid any 
biasing. 

OPBRATIOMMiIZATIOM Or THE TB8T0 

To operational i ze the test , a supply of tests were 
professionally printed: 500 copies of Form A, 100 each of forms B 
and C. Copies were made of the master tapes: 200 Form A 
(Brazilian) , 75 Form A (Lusitanian) , 50 each Forms B and C 
(Brazilian) and 30 each Forms B and C (Lusitanian) . 

A Test Manual^ giving complete information on the development, 
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uses, and administration of the PST as well as the interpretation 
of examinee scores was prepared. The most up-to-date draft of the 
manual as of the submission of this report is presented in Appendix 
H. This manual will serve as the proto-type for the manuals that 
will accompany the other semi-direct tests being developed in year 
2 of the project. An Examinee Handbook has also been prepared to 
be distributed to PST examinees before taking the test. Again, the 
most up-to-date draft of this handbook is presented in Appendix I. 
The two booklets above establish and explain in detail the 
procedures for ordering and handling the test in-house. They also 
contain registration and order forms that are used in the 
operationalization of the test. Finally, a certificate for use as 
the examinee's official score report has been designed and printed 
(Appendix J) . 

Announcements of the availability of the test have been 
produced and sent to Portuguese Language Departments and other 
xnterested parties throughout the country. in addition, a 
presentation on the test and its development has been prepared by 
Stansfield and Paiva to be given at the annual meeting of the 
American Association of Teachers of Spanish and Portuguese in 
Denver Colorado, and at the annual conference of the American 
Council on the Teaching of Foreign Language in Monterey California. 
In addition, Stansfield and Kenyon will present the psT at the 11th 
Language Testing Research Colloquium in Cuernavaca, Mexico during 
April, 1989. 



PLANS POR YEAR TWO 



The Portuguese Speaking Test development project ended with 
a wrap-up meeting that looked forward to the development of three 
additional tests in year two of the project. The semi-direct tests 
for year 2 are for Hausa , Hebrew and Indonesian • Among the 
suggestions for the next round of tests culled from the group's 
combined experience of creating and validating the Portuguese tests 
were the following (in summary fashion): 

1. Specify which linguistic/grammatical areas will be the 
most important for distinguishing between the various 
level ratings for the language and then focus questions 
to force elicitation of as much speech as possible using 
those areas. In the PST, more questions could have 
focused on the use of the preterit and imperfect, for 
example. Many examinees avoided using those tenses, so 
it was hard to know whether they could not in fact use 
them. 

2. Include more qpiestions in the "warm-up" part and provide 
a similar "winding down" part at the end of the test in 
which a few summary questions may be asked, in the target 
language, such as "How do you think the test went?" 
Include a very easy question, which everyone will be 
successful on and thus leave the test with a positive 
feeling. 

3. Eliminate the situation question in which the examinee 
has to give a toast. This question is above the 1-3 
range of the test and is outside the experience of most 
college students. 

4. Make all questions as relevant as possible to the 
examinee • s ow experience . Avoid cultural-bound 
pictures, even . : they are well known in the country of 
the target language, as they may be outside the range of 
experience of some examinees. 

Work on the next three tests has begun on a pattern similar 

to the one used for the PST described above and as outlined in the 

original project proposal. 
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PORTUGUESE SPEAKING TEST 



General Directions 



This is a test of your ability to speak Portuguese fluently and accurately. There are several pans in 
the test Directions for each pan will be given in English. Listen carefully to the directions, and 
answer in Portuguese when you are asked to do so. Your answers will be recorded on tape, so it 
will be important for you to speak clearly enough and loudly enough for your voice to be 
properiy recorded 

During the test you will be asked to put yourself in various contexts in Brazil or PbrtugaL There is 
no need to accommodate your Portuguese to the geogn^>hic location of the different questions. 
Always use the type of Portuguese you normally speak. 

Within each pan of the test, the speaking tasks you will be asked to perforai will range from easy to 
considerably more challenging. You are not expected to be able to answer all questions with equal 
facility. However, you should try to speak as much and as well as possible in response to each 
question. 

The amount of time provided for your response will vary depending on the complexity of the 
question. Since the amount of time each examinee needs to respond also varies, feel free to stop 
talking when you feel you have given a conq)lete response. On the other hand, do not worry if you 
are occasionally intemipted by the next question before completing your answer. 

Remember that there are no right or wrong answers on this test The puipose of this test is to allow 
you to demonstrate your ability to speak Ponuguese. Therefore, always say as much as you can, 
speak as well as you can and show what you can do. 



DO NOT OPEN TfflS BOOKLET 
UNTIL YOU ARE ASKED TO DO SO 
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PORTUGUESE SPEAKING TEST 



General Direcrion< 



This is a test of your ability to speak Portuguese fluently and accurately. There are several pans in 
the test Directions for each pan will be given in English. Listen carefully »o the directions, and 
answer in Portuguese when you are asked to do so. Your answers will be rsccnicd on tape, so it 
will be important for you to speak cleariy enough and loudly enough for your voice to be 
properly recorded. 

During the test you will be asked to put yourself in various contexts in Brazil or PbrtugaL There is 
no need to accommodate your Portuguese to die geographic location of the different questions. 
Always use the type of Portuguese you nonnally speaL 

Within each pan of the test, the speaking tasks you wiU be asked to perform will range fiom easy to 
considerably more challenging. You are not expected to be able to answer all questions with equal 
facility. However, you should try to speak as much and as wdl as possible in response to each 
question. 

The anoount of time provided for your response will vary depending on the complexity of the 
question. Since die amount of time each examinee needs to respond also varies, feel free to stop 
talking when you feel you have given a complete response. On the other hand, do not wony if you 
are occasionally interrupted by the next question before completing your answer. 

Remember that there are no right or wrong answen on this test The purpose (rf this test is to allow 
you to demonstrate your ability to speak Ptttuguese. Therefore, always say as much as you can, 
speak as well as you can and show what you can do. 
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PORTUGUESE SPEAKING TEST 



General EHrections 

This is a test of your ability to speak Poitugacst fluently and accurately. There aie several parts in 
the test. Directions for each part will be given in English. Listen carefully to the directions, and 
answer in Portuguese when you are asked to do so. Your answers will be recorded on tape, so it 
will be important for you to speak clearly enough and loudly enough for your voice to be 
properly recorded 

During the test you will be asked to put yourself in various contexts in Brazil or Portugal There is 
no need to accommodate your Portuguese to the geographic location of the different questions. 
Always use the type of Portuguese you noraudly speaL 

Within each pan of the test, the speaking tasks you will be asked to perfoim will range from easy to 
considerably more challenging. You are not expected to be able to answer all questions with equal 
facility. However, you should try to speak as much and as well as possible in response to each 
question* 

The amount of time provided for your response will vary depending on the conq>lexity of the 
question. Since the amount of time each examinee needs to respond also varies, feel free to stop 
talking when you feel you have given a complete response. On the other hand, do not worry if you 
are occasbnally interrupted by the next question before complecting your answer. 

Remember that there are no right or wrong answers on this test The puiposc of this test is to allow 
you to demonstrate your ability to speak Portuguese. Therefore, always say as much as you can, 
speak as well as you can and show what you can do. 
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Final Studio Script 
PORTUGUESE SPEAKING TEST 
Script-FORM A 

This is the Portuguese Speaking Test, Form A. Please silently read the general 
directions on the front cover of the test booklet while I read them to you aloud. 

[2 second pause] 

This is a test of your ability to speak Portuguese fluently and accurately. There 
are several parts in the test. Directions for each part will be given in English. 
Listen carefully to the directions, and answer in Portuguese when you are asked 
to do so. Your answers will be recorded on tape, so it will be important for you 
to speak clearly enough and IqiuUx enough for your voice to be properly recorded. 

During the test you will be asked to put yourself in various contexts in Brazil or 
Portugal. There is no need to accommodate your Portuguese to the geographic 
location of the different questions. Always use the type of Portuguese you 
normally speak. 

Within each part of the test, the speaking tasks you will be asked to perform will 
range from easy to considerably more challenging. You are oct expected to be 
able to answer all questions with equal facility. However, you should try to 
speak as mush and as ssJl as possible in response to each question. 

The amount of time provided for your response will vary depending on the 
complexity of the question* Since the amount of time each examinee needs to 
respond also varies, feel free to stop talking when you feel you have given a 
complete response. On the other hand, do not worry if you are occasionally 
interrupted by the next question before completing your answer. 

Remember that there are no right or wrong answers on this test. The purpose of 
this test is to allow you to demonstrate your ability to speak Portuguese. 
Therefore, always say as much as you can, speak as well as you can and show 
what you can do. 

[2 sec J 

Now, please check to make sure that the tape on which your responses will be 
recorded is securely threaded in the tape recorder and that your microphone is 
properly adjusted. 

16 sec. J 



Final Studio Script 
PORTUGUESE SPEAKING TEST 
Script-FORM B 



B- 1 



This is the Portuguese Speaking Test, Form B. Please silently read the general 
directions on the front cover of the test booklet while I read them to you aloud. 

(2 second pause] 

This is a test of your ability to speak Portuguese fluently and accurately. There 
are several parts in the test. Directions for each part will be given In English. 
Listen carefully to the directions, and answer In Portuguese when you are asked 
to do so. Your answers will f>e recorded on tape, so It will be important for you 
to speak ckaili enough and loudlx enough for your voice to be properly recorded. 

During the test you will be asked to put yourself in various contexts In Brazil or 
Portugal. There is no need to accomt'^'Klate your Portuguese to the geographic 
location of the different questions. Always use the type of Portuguese ycu 
normally speak. 

Within each part of the test, the speaking tasks you will be asked to perform will 
range from easy to considerably more challenging. You are not expected to be 
able to answer all questions with equal facility. However, you should try to 
speak as miuJi and as well as possible in response to each question. 

The amount of time provided for your response will vary depending on the 
complexity of the question. Since the amount of time each examinee needs to 
respond also varies, feel free to stop talking when you feel you have given a 
complete response. On the other hand, do not worry if you are occasionally 
interrupted Ly the next question before completing your answer 

Remember VvA there are no right or wrong answer^ on this test. The pu^ ^e of 
this test is lo allow you to demonstrate your ability to speak Portuguese. 
Therefore, always say as much as you can, speak as well as you can and show 
what you can do. 

12 sec] 

Now, please cUiXk to make sure that the tape on which your responses will be 
recorded is securely threaded in the tape recorder and that your inlrrophone is 
properly adjusted. 

16 sec] 
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Final Studio Script C - 1 
PORTUGUESE SPEAKING TEST 
Script-FORM C 

This is the Portuguese Speaking Test, Form C. Please silently read the general 
directions on the front cover of the test booklet while I read them to you aloud. 

12 second pause] 

This is a test of your ability to speak Portuguese fluently and accurately. There 
are several parts in the test. Directions for each part will be given in English. 
Listen carefully to the directions, and answer in Portuguese when you are asked 
to do so. Your answers will be recorded on tape, so it will be important for you 
to speak cig a rly enough and IflmUx enough for your voice to be properly recorded. 

During the test you will be asked to put yourself in various contexts in Brazil or 
Portugal. There is no need to accommodate your Portuguese to the geographic 
location of the different questions. Always use the type of Portuguese you 
normally speak. 

Within each part of the test, the speaking tasks you will be asked to perform will 
range from easy to considerably more challenging. You are nd expected to be 
able to answer all questions with equal facility. However, you should try to 
speak as mucli and as ££ll as possible in response to each quesUon. 

The amount of time provided for your response will vary depending on the 
complexity of the question. Since the amount of time each examinee needs to 
respond also varies, feel free to stop talking when you feel you have given a 
complete response. On the other hand, do not worry if you are occasionally 
interrupted by the next question before completing your answer. 

Remember that there are no right or wrong answers on this test The purpose of 
this t«st is to allow you to demonstrate your ability to speak Portuguese. 
Therefore, always say as much as you can, speak as well as you can and show 
what you can do. 

12 sec] 

Now, please check to make sure that the tape on which your responses will be 
recorded is securely threaded in the tape recorder and that your microphone is 
properly adjusted. 

16 sec.} 
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Portuguese Speaking Tesi Feedback Sbeel 
NAME TEST FORM INSTmJTION 



Thank you very much for participating in this irialing of the Portuguese Speaking Test Your 
coinmeots about the lest arc valued and will be given full consideration in the revision process 
before the final version is printed. Therefore, please answer the questions below as clearly and as 
frankly as you can. 

Part A. In the outline below, please comment on any items you felt were unclear, unfair or 
otherwise problematical for you. Especially comment on anything thai you feel interfered uith your 
ability to answer the question to the best of your present abUiiy to speak Portuguese. Such things 
might be unclear directions, unclear pictures, unrealistic situation, too Utile lime, etc 

Item Comments 



Part L 

'Conversation* 



Part IL 
Page 1 



Page 2 



Page 3 



Page 4 



1 



Page 5 



Pan III. (Page 6) 
Topic 1 



Topic 2 



Topic 3 



Topic 4 



Topic 5 



Topic 6 



Topic 7 (if applicable) 



Pan IV. (page 7) 
Situation 1 



Situation 2 



Situation 3 



Situation 4 



Situation 5 



Situation 6 
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Situation 7 (if applicable) 



Pan B. 

1. Were ihc dircaions for the different pans of the test sufficiently clear and detailed? 

( ) Yes ( ) No 

G>mment5? 



1 Would you have preferred to speak directly after instnictions to the picture-based questions 
were given rather than wit for a tone? 

( ) Yes ( ) No 

Comnients? 



3. Would you have preferred to speak directly after insiruclions to the topic questions (page 6) 
were given rather than wait for a tone? 

( ) Yes ( ) No 

Comments? 



4. Would you have preferred to speak directly after instnicUons to the situation questions (pace 7) 
were given rather than wait for a tone? 

( ) Yes ( ) No 

Comments? 
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5. Id general, were the pauses for your responses usually long enough for you lo respond as fully 
as you wished (or were %ble)? 

( ) Pauses were usually too long 
( ) Pauses were usually about right 
( ) Pauses were usually too short 

Comments? 



6. Did you feel you were unduly nervous during the lest? 
( ) Yes ( ) No 

If yes, please explain. What could have helped alleviate that nervousness? 



7. In general, did you feel that the picture-based questions, situations, and other types of 
questions in the test were adequate to probe your level of speaking abUity in Port-jguese? 

( ) Yes ( ) No 

Comment? 



Part C Please use the back of this page to comment on any aspea of the test that is not 
covered in any of the preceding questions. We would especially appredate any suggestions as to 
bow this test might be unproved Thank you very much! 
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PST Triaiing-EvaluatioD Sheet 




Directions: The Portuguese Speaking Test is intended to serve as a probe to elicit from each 
examinee a speech sample suiubie for rating on the ILR/ACTFL scale. We need sow to judge that 
the items are doing their job. As you listen to each Upc, please mark this sheet Under TIME' 
Please also indicate next to each item number whether the pause time seemed to be too short or 
too long for that examinee. Grcle 'OK.' "SH' for short or 'LO' for Long. Under 'BLOCKING.' 
comment on anything that appeared to block the examinee's response. It cx>uld be unusual 
vocabulary items, in which case write the offending word. It could also be an unclear 
undersunding of the directions or the item or that the item was clearly too difficult It could also 
be a question of time, i.e.. not enough time to think about an answer. Under 'OTHER 
COMMENTS,' describe any other ways the item tcenis to be problematical. Finally, under 
'RECOMMENDATIONS,' write any ideas that come into mind to remedy problems you have noticed. 




Portuguese Speakinf> Test Form 
Examinee's Name 


Your Name 




TIME BLOCKING 
Part L (Conversation) 


OTHER COMMEmS 


RECOMMENDATIONS 




1. OK SH LO 






i OK SH LX) 






i. OK SH L6 






T—OK — SR — ED 

5r-OK — SH — DO 












O 
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7. OK SH LO 



K. Ok SH LO 



y. OK SH — EO 



lU. OK SH — 



11. OK SH LO 



12. OK SH LO 



11 OK SH LO 



Pan II. (ficture-ba&ed questions) 
1. OK SH LO 



1 OK SH LX) 



3. OK SH LO 



4. OK — SH — DJ- 



5. OK SH LO 



fan 111. (Ass^^ers lo questions) 
1. OK SH LO 



I UK SH — UT 



J. OK SH LO 
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4. OK SH LO 



b. UK SH — LD" 



6. OK SH LO 






7. OK SH LO 






Pan IV. (Situations) 
1. OK SH LO 






^ UK bH UJ 






J. UK Ml LU 






4. OR SH LO 






5. OK SH LO 






c>. OK SH LO 






7. uK iH LU j 
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APPENDIX E 



PST TriaUng-Rater Evaluation Sheet 



The Portuguese Speaking Test is intended to serve as a probe to cbcit from each examinee a 
speech sample suitable for rating on the ILR/ACTFL scale. For each eaxmince, as you listen to the 
tape, please show on a scale from 1 to S (with 1 being the lowest and S being the highest) how 
each item contributed towards forming a rauble speech umple for that examinee. Please also 
indicate next to each item number whether the pause time seemed to be too short or too long for 
that examinee. Qrcle 'OlC'Shorf or "Long.* If any item seemed problematical for the student, 
please comment under the item number. 

Portuguese Speaking Test Form 

Examinee's Name 



P «= item was problematical (please ooniincoti 

1 B item did not help in making a rating 

2 « item keiped lomewhat in making a rsting 

3 B item was lucful in making a rating 

4 m item was vcfy useful in making a rating 

5 m item was extremely useful in making a rating 

NA * item was not used or not used in rating (e.g. warm-up items) 

Part L (Conversation) 



1. OK SHORT LONG 



NA 



1 OK SHORT LONG 



NA 



3. OK SHORT LONG 



NA 



4. OK SHORT LONG 



P 1 2 3 4 5 NA 



5. OK SHORT LONG 



P I 2 3 4 S NA 



6. OK SHORT LONG 



P 1 2 3 4 5 NA 



7. OK SHORT LONG 



P 1 2 3 4 5 NA 



8. OK SHORT LONG 



P 1 2 3 4 5 NA 



9. OK SHORT LONG P 1 2 3 4 5 NA 

10. OK SHORT LONG P 1 2 3 4 5 NA 

11. OK SHORT LONG P 1 2 3 4 5 NA 
11 OK SHORT LONG P 1 2 3 4 5 NA 
13. OK SHORT LONG P 1 2 3 4 5 NA 
Part 11. (Picture-based questions) 

1. OK SHORT LONG P 1 2 3 4 5 NA 

1 OK SHORT LONG P 1 2 3 4 5 NA 

3. OK SHORT LONG P I 2 3 4 5 NA 

4. OK SHORT LONG P 1 2 3 4 S NA 

5. OK SHORT LONG P 1 I 3 4 5 NA 

Pan IIL (Answers to questions) 

1. OK SHORT LONG P 1 2 3 4 5 NA 

1 OK SHORT LONG P 1 2 3 4 5 NA 
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3. OK SHORT LONG 

4. OK SHORT LONG 

5. OK SHORT LONG 

6. OK SHORT LONG 

7. OK SHORT LONG 

Part IV. (Stiiutjons) 

1. OK SHORT LONG 

1 OK SHORT LONG 

3. OK SHORT LONG 

4. OK SHORT LONG 

5. OK SHORT LONG 

6. OK SHORT LONG 

7. OK SHORT LONG 

ERIC 



P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 5 NA 

P 1 2 3 4 S NA 

P 1 2 3 4 5 NA 

F 1 2 3 4 5 NA 
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Questionnaire for Partiqunts in the Portuguese Speaking Test Study 

We would very much appreciate your answers to the following questions concerning the three 
speaking tests (live interview plus two tape-recorded tests) which you took in connection with the 
Ponuguese Speaking Test Study. We are asking for your name at the end of the form simply as a 
check that the questionnaire has been received. AU information will be analyzed on an anonymous, 
group basis for purposes of test improvement. Please feel free to make any comment you wish in 
response to any question in the space provided or on the back. Thank you in advance for your 
comments and for your much-appreciated participation in the test development process. 

(1) Over the course of the live inffrvifw. do you feel that your maximum level of 
speaking ability in Portuguese was adequately probed by the tester? 

( )Yes ( )No 

CommcnLs: 



(2) Over the course of the taned fxf. do you feel that the descriptions, narratives, 
situations, and other types of questions in the test were adequate to probe your 
maximum level of speaking ability in Portuguese? 

( )Yes ( )No 

Comments: 



GO 



(3) In (he live infervifw. were there any questions asked or speaking situations 
required which you felt were in any way 'unfair*? 



( )Ycs ( )No 

Comments: 



(4) In the taped tests, were there any picture/descriptions, narratives, situations, 
or other questions that you felt were in any way 'unfair'? 

( )Ycs ( )No 

Comments: 



(5) Ip which of the two types of test (live interview or taped test) did you feel 
more anxious or nervous? 

( ) Live interview 
( ) Taped test 
( ) Both the same 

Comments 



CI 



(6) Which of the two types of test (live interview or taped tes«) did you feel was 
more difficult? 

( ) Live interview 
( ) Taped test 
( ) Both the same 

Commgnts: 



(7) In the taped test, were the pauses for your responses usually long enough for 
you to respond as fully as you wished or were able? 

( ) Pauses were usually too short 
( ) Pauses were usually about right 
( ) Pauses were usually too long 

Comments: 



(8) Where the directions on the taped test clear? 

( ) Yes ( ) No 

Comments: 



C2 



(9) Which of the two types of test did you prefer-the live interview or the taped 
test? 

( ) Live interview 
( ) Taped test 
( ) No preference 

Comments: 



(lO)Please use the space below to comment on tny aspect of the live interview or 
taped test that was not covered in any of the preceding questions. We would 
appreciate any suggestions as to bow these testing procedures might be im- 
proved. 



Thank you vciy much for youi help. 
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Comments from the Participants in the 
Portuguese Speaking Test Validation Study 

Below are all the comments written on the questionnaires that were completed by all 
participants in the study. The comments are copied below as written on the questionnaires. 
No attempt has be made to correct grammar, etc. Any necessary clarification that has been 
added is written in parentheses. 



Question ]: Over the course of the live intervi ew, do you feel that your maximum level of 
speaking ability in Portuguese was adequately probed by the tester? 

She (the tester) was vciy efficient and thorough. 

Much easier since we have time and non-verbal communication. 

Speaking to the tester, I was more at case and she could gear the questions to my ability. 

p^es,) however, in the beginning it was a little bit difficult. I do extremely well in an 
informal setting. After a while though, I didn't feel so much pressure, and felt the 
conversation flowed a bit more freely. I lw£ to speak Portuguese and take any chance I get 
to practice it. 

It was such a difference and such a relief to be talking to a real person. It took me a while 
to get used to the recorder. I thought she did a pretty good job seeing what I could and 
could not do. 

I most likely felt comfortable with the interviewer. 

I was a little niervous. But I tried to do the best I could. 

Somehow I was a little nervous during the first part of the live interview, but the person to 
whom I was talking was very friendly and that helped me a lot. 

For the most part, I felt that the 1/2 hour interview was adequate, but I think an hour 
would be more appropriate. Also, some more creative questioning would be better. 

In a real conversation with another human being, communication flows naturally and one 
easily reaches the outer bounds of one's abilities. 

Maximum no, closer to 80-90%. The reasons being that I was a little uncomfortable and 
when I was lacking just 1 word I sometimes couldn't finish the idea I was trying to get across 
and I was a little nervous. The; are the reasons why my maximum level was not achieved. 

My ability to speak wasn't tested ^ell, because I did not understand the interviewer very 
well. 
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I feel I could have been pushed a little farther (although I did have some vocabulary 
trouble). 

I think that different situations and different subjects make proficiency vary. 

Not maximum ability because it needs to be longer. One does not get into a complete 
Portuguese state of mind in 5 minutes. I was still thinking in English. 

We had an interesting interchange, although she didn't test specific vocabulary areas. 

Some of the topics of conversation were rarely used and required a vocabulary suited for 
it 



Question 2' Over the course of the taped test do you feel that the descriptions narratives, 
situations and other types of questions in the test were adequate to probe your maximum 
level of speaking ability in Portuguese? 

I thought the tape test was very good. 

The variety was excellent. 

I think the different situations that the test presents to us are very adequate, because they're 
about common situations in real life. 

The situations probed all different types of language and vocabulary. 

Obviously, these kinds of tests have limited coverage and probe to examine the full capacity 
of someone's language skill. Anyway, it was reasonably good, but not better than live 
interview. 

I think that the potential for creativity allowed me to reach the maximum level but only 
because not every one is creative in the same way. 

It's relative what one can answer with the time constraints but I still think that what the tape 
has is the best that can be done to test non-native speakers. 

Giving directions was the hardest part to describe. 

The situations were adequate, but the same test should be distributed by a live interviewer. 

It's difficult to be creative when the situations seem artificial, contrived. It is difficult to 
communicate one-way-with no person to respond to, etc. 

I felt very uncomfortable speaking into the microphone and this contributed to not probing 
my maximum level. Again, when I was caught up with one word it was difficult to continue. 

They went far beyond my capabilities. 
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Actually, they were somewhat above my level, perhaps too much. 

I felt very pressured and did have a bit of difficulty hearing the speaker on the tape. I feel 
my Portuguese speaking ability is quite good especially after living in Brazil only one year. 
IVe been in a lot of situations already described in the tests but I feel it's easier to talk 
about them when you are actually in the situation. 

I found it difficult at times to judge the pause (i.e. what really is 1 min. 45 sec.) At times 
I felt restricted by the pictures themselves. 

Although I didn't think this was quite enough hypothetical stuff to really p'obe for a ceiling, 
but maybe those couple were enough. Otherwise yes ( = taped test probed maximum 
level). Also, there wasn't very much narration in the past. 

I felt I was being tested on aspects of the language that are not emphasized at Georgetown, 
while subjects such as politics and culture were largely omitted. We spend far more time 
on topics such as Brazilian race relations and foreign debt than we do on water skiing and 
sightseeing. In short, I was tested on things I didn't know how to say, and was not tested 
on what I did know how to say. 

(Thinks it was probed adequately) Although in the end I had a headache because of the 
length of the test. 



Question 3: In the liv e interview , were there any questions asked or speaking situations 
required which you felt were in any way 'unfair^ 

Every question asked to me was very good. 

No questions seemed unfair. 

Questions were rephrased when I didn't understand and the tester was able to see my 
limitations and not push me. 

Unfair-no; after I got accustomed to it, I felt everything flowed easier. 

The social situations (in my case ordering my secretary around and arguing with a store that 
sold me a watch) were unfamiliar to me even in my own country. I am a student, so I'm not 
used to issuing orders to anyone, and I don't own a watch. The rest of the live interview 
involved questions for which I was better prepared. 

The only problem I experienced was that the Portuguese I have learned is ft^om Brazil, and 
the person I had the interview with was from Portugal so we had some regional terminology 
differences. 

A couple of times I did not understand the interviewers "portugal" accent 
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Question 4; In the taped tests^ were there any picture/descriptions, narratives, situations, 
or other questions that you felt were in any way ♦unfair*? 

They are appropriate for testing ability. 

Maybe the ones that require mo«-e cultural knowledge. 

I didn*t like veiy much the question about labor workers and what I think about syndicates. 

Inappropriate perhaps. Having to complain to a neighbor is something I would be unlikely 
to do. A party at 2 am is not unusual in Latin America and threatening to call the 
authorities might be met in derisive laughter. 

If I had a greater knowledge of vocabulaiy I may have been able to answer them. 

The directions were difficult to give properly in the time allotted. 

Not unfair, just difficult. There is no way to prepare for a test like this. I undentood 
everything, but I feel once faced with any of the situations in reality = I'd be able to put 
more of myself into it ... "restaurant scenes, etc." 

(No) Although I find some of the diagrams very limiting. 

Some of the pictures were too detailed and inadequate time was given to properly describe 
them. 

See #1 Again, the taped test seemed more appropriate for someone who had either Uved 
in a Portuguese-speaking countiy, or had spent their class time preparing for that. 

I think there were some conjuring pictures that I could not have done better describing them 
in my own language, but generally they were fair. 

Please see # 10 for comment. 

No 10: I would like to offer the following comments on the taped test: 

- 1 thought that the situational questions were excellent. They were typical situations that 

make one think fast and use different types of language. 

- I found some of the topical questions (e.g. gun-control and bilingual education) very 
difficult to answer given the short amount to time allotted for formulating the answer. In 
general. It seemed that less creative people • or those less familiar with those specific issues 

- might not be able to demonstrate their maximum level of speaking ability on those 
questions. 



QgestjQn g; in which of the two types of test (live interview or taped test) did you feel 
more anxious or nervous? 



I did not feel nervous or anxious at all. 



Wasn't nervous in either task* 

It (the taped test) was harder I believe because of the time constraints. 

Being recorded and responding in a fixed time frame creates pressure. 

I find it very uncomfortable to engage in a conversation with a machine! 

I did feel a bit nervous perhaps mostly because of the time limit. When I really get the 
change to speak in Portuguese I do very well and I felt that on the tape that I didn't get the 
opportunity to show all I know Perhaps it was because of the "nervousness." 

It was hard tc get used to the tape. And getting cut off, or finishing to soon and waiting is 
so annoying. Voice activated recorders maybe? Or is that too expensive? I hi^y 
recommend it for the real t*iing. 

Because of time limitation the taped test made me more nervous; there was no feedback. 
The beeping noises get on my nerves. 

Tm not accustomed to making speeches without interaction in conversation. When I do 
have to make speeches, I invariable have more time io prepare my remark and tailor them 
to the allotted time. Furthermore, I never have to make speeches in Portuguese and don't 
foresee that possibility in the near future. 

There is moie time flexibility and real communication with a live interview and that made 
me more relaxed and comfortable speaking in Portuguese. 

I deal better face to face. Also, I lost some interest in speaking to a ;ape recorder. 

V^Tien speaking to a person, one feels more comfortable and relaxed as opposed to the 
taped test. 



Question 6: Which one the two types of test (live interview or taped test) did you feel was 
more difficult? 

Both seemed fair. 

Duri^ig my live interview I much talked about themes that I wanted to and that I like. 

(The taped test) Only because of the time requirements. 

The tape required much more vocabulary and social knowledge. 

Again, talking to the machine makes me uncomfortable. Did not use any full potential. 



It was very tedious, although quite complete. 

The taped test was definitely more "difficult." 

I personally do not feel relaxed when speaking entirely to a machine. 

Mostly because I felt more pressured, and it was z very passive kind of thing-all these 
things in front of your eyes to describe. 

(The taped test) Because of limited time. 

Question 7: In the taped test, were the pauses for your responses usually long enough for 
you to respond as fully as you wished or were able? 

Gave ample speaking time. 

Several times, however, I did get cut off on the last part. 

On the long worded" questions the time given to "prepare" to answer the question was 
short. 

I did however, have to push myself to fill space. Conversation with a machine just doesn't 
animate one to chatter much. 

For the most part, they were good. However, I did feel that some pauses were too short. 
In some situations I wouldVe liked to say mere. Some areas definitely need more time for 
us to be able to speak more fi-eely and creatively. 

Both - 1 got it just right sometimes. It's hard to judge how long to take, or how long to give. 

Pauses were usually either too short or too long. 1 know that sounds cryptic, but that is what 
I found. 

I feel shorter pauses would be better - particularly for the less fluent speaker. Shorter 
pauses would keep things moving and thus the interest level would stay higher. Also long 
pauses may test an individual's planning ability - not necessarily their speaking ability. 

Question 8: Were the directions on the taped test clear? 

Very clear. 

Repetitious. 

Too clear. I felt as if they think everyone who takes the test is a moron. The directions can 



be speeded up and shortened. 

Too long and repetitious - we don't need them afier each scenario - especially the last page. 



Qgestjon 9; Which of the two types of test did you prefer-the live interview or the taped 
test? 

While the live interview dealt with relatively familiar topics, the taped test was more 
interesting and challenging because it incorporated various types of testing-approachen. 

I felt kind of retarded doing the taped test. It wasn't that bad though, and the situations 
were rather interesting ~ good questions. However, that getting cut off was so annoying!! 
Ugh. 

(The live interview) Definitely a better way, and more natural of examining ability. 

I feel the live interview gave me a bit more of a chance to show my Portuguese speaking 
ability. I felt much more relaxed with it iLan with the tapes. 

(The live interview) Being able to use hand gestures really helped me make myself 
understood. 

Again, it (the live interview) is far more realistic, and reflective of my training. 

(It the live interview) By hearing a live voice made conversing much more conducive. 



Question 10: Please use the space below to comment on any aspect of the live interview 
or taped test that was not covered in any of the preceding questions. We would appreciate 
any suggestions as to how these testing procedures might be improved. 

I think both the live interview and the taped test were very good organized, because it gave 
me the opportunity to measure myself with the Portuguese language. 

I think that the testing procedures are well done as it is and I would not change them. 

I consider that the live interview is more important and the taped test because you can see 
the reaction and gestures of the per'.in with who you are talking to. I would prefer the 
whole research in a form of live interview. 

The live interview was a better measure for me. The fact that I could ask if I didn't 
understand something helped. I found the tape test "issponse" times to be too shon. Also 
the issues on some of the questions (labor unions for example) were topics that couldn't 
answer well even in English. 

A tape is not - proper substitute for a live interview with personal interaction. 
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I feel to a certain extent, the more "informal (not in terms of grammar or anything like 
that) the easier it is to speak TIet yourself go" as it were. I did enjoy taking the tests though. 
Also I felt with the taped tests I was more limited in what I could say. I felt that I was only 
"allowed" to talk about scenes in the pictures and that I had to make sure to mention 
everything. This was difficult I felt veiy limited and this made me a bit anxious. In a live 
situation, it is easier to show one's Portuguese ability. 

The role playing was also somewhat difficult. I generally do not speak freely in contrived 
situations. 

The testing procedures were reasonably good enough to examine someone's language skill 
in some extent 

Examples should be given of how you are supposed to respond. 

Put briefly, I think you have to test people in roughly the same way you educate them. This 
applies to the curriculum as well as the pedagogical method. 

It is my personal opinion that in order to have a more realistic idea of the level of 
proficiency in a foreign language is at most a written test and live interviews instead of 
taped tests. 

Try to pick topics and issues more current and applicable, not to say that these were not, 
but these topics were a bit awkward. 

In the live interview, any off limit subjects such as the interviewer can't translate any words 
for you. 
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PREFACE 



Portuguese Speaking Test Manual has been prepared for all who are interested in the 
Portuguese Speaking Test (PST): for individuals considering using the test, for those who 
administer it and for those who use oral proficiency ratings based on the test. In addition 
to giving detailed instructions on test administration, the Manual gives an overview of the 
PST program, information on the development, form and content of the test, notes on 
interpretation of the test results, and technical information about the test 

This is the first edition of the Manual. Your suggestions for improving its usefulness 
are most welcome. Please feel free to send any comments to us. 



IMPORTANT NOTE TO TEST SUPERVISORS 

This manual contains import ant in formation about the procedures that must be followed 
BEFORE, DURING, and AFTER the administration of the Portuguese Speaking Test. 
Uniform procedures are essential if a standardized test such as the PST is to yield consistent 
results. The scores of examinees from vzrious institutions in the nation will be comparable 
onfy if all supervisors follow the same procedures and give exactly the same instructions. 
It is necessaiy, therefore, that you read the entire section entitled Test Administration" 
before administering the test and follow the instructions without exception during the test 
administration. 
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OVERVIEW OF THE PST PROGRAM 



General Infonmatinn 

The primaiy purpose of the Portuguese Spr^king Test (PST) is to evaluate the level of 
oral proficiency in Portuguese attained by Amencan and other English-speaking learners 
of Portuguese. It is intended for students at proficiency levels from Intermediate-Low to 
Supcnor according to the proficienc y guid elines developed by the Amencan Cbuncil on the 
Teaching of Foreign Unguages (ACTFL, see pages 15-18) or alternatively from 1 to 3+ 
on the scale of the interagency Language Roundtable (ILR) of the VS. government. Each 
examinee receives a rating on a modified ACTFL/ILR scale. 

The PST is designed to em ulate the direct face-to-face oral proficiency interview used 
by goyemment agencies and ACTFL through a semi-direct format In semi-direct testing, 
cxaminets are presented with the stimulus through a test tape and a test booklet; their 
responses are recorded on a second tape. This PST examinee response tape is then sent for 
outside ratmg by specially tniined raters proficient in using the ACTFL/ILR scale. Semi- 
du-ect testing is not intended to replace the oral proficiency- interview. However, it is 
intended to offer a standardized approach to oral proficiency testing in situations where it 
is not feasible to give an oral imcrvicw due to the unavailability of trained testing personnel, 
or lack cS time and budget for such testing. 

The PST may be administered in a language laboratoiy fretting and is thus ideally suited 
to laige group testing. Alternatively, the test can be administered individually using two 
tape recorders. 

Test Format 

There are three components to the PST: the master test tape, the examinee test booklet, 
and the examinee response tape. 

The master test tape contains the directions and test questions. All directions are given 
in English. In order to ensure that Portuguese speaking skills are tested (rather than 
listening comprehensionX «W questions are in English, with the exception of the Vann-up" 
questions in Part One (see bckiw). 

The examinee test booklet contains the general directions on the front cover, five pages 
of drawings for the picture-based questions, and two pages of written questions. 

The examinee resposse tape is a blank tape on which examinee responses are recorded 
dun'ng the administration of the test 

There are three parallel forms of the PST. The master tape is available in either a 
Brazilian or a Lusitanian version. 




T«t Content 



The entire test lasts appronmately 43 minutes and elicits a speech sample from the 
examinee of approximately 20 minutes in length. Six types of questions are used. 

1. Personal Conversation. 

This part corresponds to the "warm-up" used in the direct interview. In this section, 
the examinee listens to short questions about his/her family, education, hobbies, etc. in 
Portuguese and responds to each question as it is asked. There are either 12 or 13 such 
questions on each form. This is the only section in which Portuguese is used on the tape. 
After each question is asked, the examinee has between 5 and 25 seconds to respond, 
depending on the complexity of the question. 

For each question in parts 2 • 6, the examinee is given between 15 and 30 seconds to 
prepare an answer before being required to speaL Time for giving an answer varies from 
45 seconds to 1 minute and 45 seconds, depending on the complexity of each question. 

2. Giving Directions. 

The examinee is shown a pictorial map in the test booklet and is instructed to give 
directions between two points on the map. The question is contcxtualizcd in that examinees 
are instructed that they are giving this information to a friend over the phone. 

3. Detafled Description. 

The examinee is shown a drawing in the test booklet and is instructed to describe the 
picture in as much detail as possible. Each picture depicts a variety of objects and actions. 
This part is contextualized so that the examinee knows the specific audience being addressed 
and the purpose of the description. 

4. Picture Sequences. 

The examinee is instructed to speak in a narrative fashion about a sequence of four or 
five pictures shown on a single page in the test booklet There are three questions of this 
type; one each designed to elicit present, past and future time narration. All questions are 
contextualized so that the examinee is given a specific audience and a specific reason for 
the narration. 

5. Topical Discourse. 

The examinee is instructed to talk about selected topics invohang different discourse 
strategies. These strategies include explaining a process, supporting an opinion and talking 
about a hypothetical situation. The examinee hears and reads in the test booklet five such 
topics. Following are examples of these questions from the PST Ficaminee Handbook . 
These are examples and not actual questions used. 



Please describe the lands of leisure-time activities you particularly enjoy. 

Talk about the advantages and disadvantages of using public transportation. 

Explain to a Portuguese-spealdng friend how one would go about buying a used car in 
the United States. 

Some Americans feel that foreign language education should begin in the first grade 
or even kindergarten. Do you agree or di:>agree? Explain why. 

If you were financially able to spend one year traveling, free of any other 
responsibilities, where would you go and what would you do? 



6. Situations. 

The examinee hears and reads a printed description of a real-life situation in which a 
specified audience and communicative task are identified. The examinee is then instructed 
to carry out the specified tasL The tasks range from making simple requests to giving a 
formal toast Below are examples of these questions from the PST Examinee Handbook . 
Again these are examples and not actual questions used. 



You are at a hotel in Brazil. Tell the clerk yoc need a double room with private bath 
for two nights. Ask about room rates and if you can pay by traveler's check. 

You are at a clothing store in Lisbon. Ask if you can return a shirt you bought there 
last week as you discovered it had a stain on it when you brought it home. 

You want to buy a ceramic bowl at a stand at an open air market in Lisbon. However, 
you feel the asking price is too high. Tiy to convince the shopkeeper to lower the 
price. 

When you arrive at the airport in Rio de Janeiro, you discover your luggage was sent 
to Europe. The clerk tells you it wiU be returned in three days. Irate, you ask the 
airlines to give you money for the new clothing you will need to buy for your important 
business meeting tomorrow. 

After staying with a large Brazilian family for the summer, you take them out to a 
formal restaurant to express your thanks. Before the meal begins, you formally give 
a little toast expressing your thanks for the family's hospitality during your stay. 
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Usesofth^PST 



In keeping with its intended function as a measure of oral language proficiency, the 
primary focus of the PST is to assess the abiht y to s peak Portuguese in con emporary, real- 
hfc language-use contexts according to the ACTFL scale. Such a rating i ly be useful in 
a variety of contexts and for a variety of purposes, such as: 

• admission to a Portuguese study program 

• placement within a Portuguese study program 

• exemption from a Portuguese language requirement 

• application for scholarship or appointment 

• competenqr testing upon exit from a Portuguese program 

• certification of Portuguese language proficiency for occupational purposes 

• evaluation of a Portuguese instructional program 

The PST program encourages use of the test by an institution if such use will assist it in 
making valid decisions in one or more of the above applications. However, the institution 
Itself must determine whether the PST is an appropriate test with respect to its intended 
use. 



Local Validation and Standard Sfftinf 

After it has been decided that the PST is appropriate for a particular use, it may be 
fitting to conduct a local validation study if an important decision will be made on the basis 
of the PST score. Such may be the case if, for example, the PST is used to determine 
suitability for a particular job that requires language skills. In such a case, one might want 
to conduct a research study that would identify whether a relationship existed between 
performance on the PST and success on the job. If through research such a relationship 
were found to exist, then the study would provide evidence of the PST for that particular 
use. 



In addition to establishing the validity of the PST for a particular use, it may also be 
appropriate to justify any particular criterion of minimum performance on the PST that has 
been set Such a study, known as a standard setting stu<fy, can provide data tJ.at may be 
used to validate an existing standard or establish a new standard of acceptable performance 
on the PST. For examples of such studies in the context of a speaking test and a discussion 
of appropriate methodology see Livingston (1978) and Powers and Stansfield (1985). PST 
program staff will be pleased to advise an institution on how to conduct an appropriate local 
validity and standard setting study. However, the PST program can not provide financial 
support for such studies. 



TEST ADMINISTRATION PROCEDURES 



This section (pages 5-15) provides crucial information directed to test center 
supervisors. It discusses PST operational proccd jres and includes step-by-step 
instructions for administering the test. It is absolutefy essential that these directions 
are read and followed exactly by all PST test supervisors to ensure the validity of PST 
scores across test centers. 



GENERAL INFORMATION 
Cbrnmunicarion with TAI. 

Most communication between test center supervisors and CAL can be handled by mail 
which should be addressed to: 

Coordinator, Portuguese Speaking Test 
Center for Applied Linguistics 
1118 22nd Street, N.W. 
Washington, DC 20037 

Sometimes, however, it may be necessary or more convenient to contact CAL by 
telephone. Please direct all phone calls to CAL at (202) 429-9292, asking for the 
Portuguese Speaking Test Coordinator. CAL cannot accept collect calls. 

Registering for the Tert 

Registration for the PST may be initiated by institutions or by an individual examinee. 

Ipstitvtipn?! Rtgi?tr?tjQn> An institution may register to administer the PST to its students 
on dates that are convenient to the institution. To place an order for the test, fill out a PST 
Wcr Fpnn . (A copy of the PST Order Form is printed on the last page of this Manual . 
Additional copies may be obtained from CAL or photocopied from this Maimal ) Follow 
the instructions for completing this form below. All orders for the PST must include this 
Wcr Form, whether testing is initiated by an institution or by an individual examinee. 

1. Type or print ail information clearly. 

1 Fill in the shipping and billing information. Do not forget to include appropriate 
lelephooe Humbers. 

3. Check whether this is your first order. If it is, CAL reserves the right to verify all 
information on the form before sending test materials to you. 

4. Mark the test form desired. In most cases this vkrill be Form A, unless examinees are 
being retested or have already used Form A. If your institution regularly administers 
the PST, you should alternate the form of the test being used with each 
administration. Use a separate order form for each test form ordered. 
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5. Mark the Upe version desired. A'»hough aP of the PST directions and most of the 
questions are u English, Portugi j is used . . the "warm-up" section (sec question 
type 1, p. 2). Thus, you shou! ;der the tape version (Brazilian or Lusitanian 
Portuguese) that is most appropi,ate for your examinee(s). Use a separate order 
form for each version if both versions are being requested. 

6. Fill in the number of •um:oces expected, testins date and nquirtd deliveiy date. 
Please note that a PS i order should arrive at the PST Program Office at CAL at 
least 2-3 weeks aefou, the intended testing date to allow timely shipment of the 
necessary test materials. Telephone orders are accepted only under urgent 
circumstances and only from institutions with which CAL has established previous 
contact 

7. Include information o;; the purpose and nse of the test (optional). 

8. Read the information on payment carefully. It is your responsibility as test supervisor 
to collect payment from the examinee(s). You will be charged $60.00 per examinee 
tested. (Note: You may collect more from each examinee to cover any testing costs 
incurred by the institution in administering the test. Examinees should be told in 
advance the amount of additional charges. This information may be written on the 
front cover of the Examinee Handhonk;> 

9. The security asreement between the test supervisor (the person actually responsible 
for administering the test) and CAL is most critical. The security afr-tement must 
be signed before any materials will be sent out. All first orders must be -nade using 
this form. Subsequent orders, under urgent circumstances, may be made by 
telephone by test supervisors who have submitted this form. Be sure to include your 
tifnature, the date, and print your name, telephone number, title, and department 
in the spaces provided. 

10. Mail the Order Fprni« along with the Individual Registration Fnrni if applicable, to 
the address above. 

Upon receipt of the Order Form. CAL will forward to the test center supervisor the 
required numl^r of PST test booklets and associated materials, including the master tape(s) 
and a back-up of the test tape(s), blank cassettes for examinee responses, copies of the 
faamincc HandhOOK. and necessary forms. The Examinee Handhnnj^ contains important 
mformation to prepare examinees for the PST and should be given to the examinees well 
ahead of the testing date. The front cover of the handbook includes space to write 
important test information for the examinee, such as date, tim , location of the exam, etc. 
You may want to fill this out before you distribute the handbooks lo the examinees. 

It is your responsibility to organize the \oc*.l gistration of examinees. When an 
examinee registers locally for the PST, he/she must present an acceptable form of photo 
Identification (e.g., driver's license, $tudent ID, or passport). You must verify the 
Identification and record the examinee's name and social security number. Examinees are 
required to bring this photo identification with them to the test, where it should be iouble- 
checked. 

Individtia! Regi^trgtion. You may be asked to serve as test supervisor by a potential 
examinee who wishes to take the PST. In this case, that individual should have followed the 
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procedures indicated below. These procedures are explicitly described in the 1 'minee 
Handbook . 

1. The individual contacts CAL for the name of the nearest PST test center. 

2. If a nearb)- test center is available, the examinee then personally contacts the test 
center supervisor. 

3. If a nearby test center is not available, the examinee contacts a local institution that 
offers a Portuguese language program. If no local institution offers Portuguese, the 
examinee contacts the director of the institution's language laboratory, or if none is 
available, local foreign language teacher who is willing to administer the test. 

4. The examinee pays the test fee ($60) via personal check or money order made out 
to Portuguese Speaking TestlCAL directly to the test supervisor. This payment roust 
be included with the PST Order Form . 

If you have been asked to serve as test supervisor for a single examinee, you need to fill 
out the PST Order Form as instructed above, and send it to CAL, tc^ether with the 
naminee*! IntfivjdMal Rtff?trifjpn Form. In this case confirmation of the examinee's 
Identification is required on the Individual Re pistration Form Also, you may want to 
admmuter the KT using the INDIVIDUAL ADMINISTRATION instructions beginning 
on page 15 of this manual. All other information in this section still pertains to you. 

Test Fct 



The fee for the PST is $60 per examinee. For individual registration, the examinee's 
payment in the form of a check or money order made out to Portuguese Speaking TestlCAL 
must accompany the PST Prdgr Form (see above). For institutional registration, test fees 
should be collected fi-om the examinees eit'ier at the time of registration or . n the testing 
date. No e aminec should be allowed to take the PST without first prci.nting the fee to 
you. The testiijg fee can be paid by the institution by check payable to Portuguese Speaking 
Tat/CAL. It is preferable for payment for multiple examinees to be made via a single 
check. If a purchase order is used for payment, it must accompany the PST Order Fnrm . 
In this case, an invoice for the total testing fee, including any delivery charges for rush 
orders, will be issued to the institution after the test has been administered and the test 
materials have been return to CAL This invoice should be paid by check payable to 
Portuguese Speaking Test/CAL. ' 

CAL does not provide an honorarium to tesf center supervisors. If an institutional fee 
for administering the test required, you should collect it directly from the examinees 
tested at the institution. You are responsible fo' forwarding to CAL only the established 
test fee of $60 per examinee. 

Test Segurifv 

It is extremely important that the PST be safeguarded and administered under secure 
conditions at each partidpating institution. By agreeing to ser.e as the supervisor of the 
test, you assume the responsibility of enforcing test security from the time you receive the 
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test materials until you return them to CAL It is, therefore, necessary that you adhere to 
the specific instructions below. 



1. Keep all test materials either in your immediate physical possession or in a locked 
cabinet or other i^cure area under your control. 

2. Do not copy, or allow others to copy, any portion of the test booklet or tape, or 
make any notes or transcriptions of the test booklet or tape content 

3. Allow only those particular individuals who are to be tested to sec the test materials, 
and only at the time of test administration and under the specific procedures 
described in this manual. 

4. Sijould any irregularities occur, report them in the Supervisor's Report Form 
included in the test package. 

5. Make sure you collect all test materials from the examinees after administerine the 
test. 



Your serving as test supervisor constitutes an implicit agreement to conform to the 
above conditions. Your cooperation in this matter is absolutely essential. 

Score Renortiny 

Approximately 3 weeks following receipt of the test materials, CAL will forward to 
the institution a score repon containing, in roster form, each examinee's rating. In addition, 
official score reporu will be sent to examinees and to as many as two other addresses they 
indicate on the Request Form for Official Scnr«' pnrtc This form is located on the back 
cover of the Exammee Handbook. These forms should be collected from the examinees on 
the day of the test and returned to CAL by the supervisor with the PST test materials. Note 
that under individual registration, a score repcn roster is not sent to the test supervisor 
unless requested by the examinee. 

The ofiRcial score repon Lidicates the name of the examinee, the test date, test center, 
and score on the test. For purposes of security, the score repon is embossed with the 
irreprodudble raised seal of the Center for Applied Unguistics. The reverse side of the 
score report contains a description of the PST scale for score users. 

TestCeiit>rCode« 

Each partidpating institution is assigned a three-digit test center code for data 
processing purposes. Test center codes can be found on the top of the Test Materials 
ChcckllSl enclosed in the package of test materials. This information is needed when 
labeling the tapes containing student responses. 

Receipt of Test M.teri.N 

Immediately after the shipment of test materials is received, the contents should be 
verified against the Test Materials Checklist. If there is any discrepancy CAL should be 
contacted immediately for replacement of materials. 
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The test materials must be placed in a securely locked area. No one but the test 
supervisor, a designated associate or responsible official should be allowed to have access 
to the storage area (see Test Security above). 

Checking the Condition nf T est Miteriah 

Although CAL makes cveiy effort to ensure that test materials are in good condition 
before they are shipped, it is still pcasMc that some defects may escape attention. It is 
therefore important that you double check the condition of the test booklets and test tapes 
upon receipt of the materials. The following steps are recommended: 

1. Check to make sure that all test booklets are properly scaled. 

1 Make sure that the test tapes are in working order. (Note that there arc many 
pauses in the test tape.) Be sure to rewind the tapes completely after you check 
them. r J J 



Distribution of F»flmm*T Pandbool« 

In order to prepare examinees for the PST, distribute a copy of the Examinee Handhonk 
to each examinee weU ahead of the testing date. You may want to write the specific test 
uifonnation your examinees need to know (exam date, time, location, etc.) on the form 
pnnted on the front cover of the Handbosk before you distribute them. (Note: The 
faamincc Handbook may be photocopied for other interested individuals if desired.) 



nie Testinf Site 

It is your responsibility to arrange for a testing site. The PST may be administered in 
either of two ways: 

1. 1.1 a language laboratory capable of huving students listen to a master program 
distributed from the console, while recording their responses at the individual booth 
positions. This is most appropriate when two or more examinees are being tested 
at the same time. However, it is also the method of choice for a single 
administration as well. 

2. In a regular classroom or other quiet location, using TWO cassette recorders. One 
recorder is used to play the master test tape 'using the PLAY mode) and the other 
recorder is used to record the student's responses (using the RECORD mode). It 
u not possible to give an individual administration of the PST using one cassette 
recorder only. Under these circumstances, only one student can be tested at a time 
(unless there is more than one set of recorders and testing rooms). This procedure 
IS appropriate when a single examinee is being tested or there is no other alternative. 



Please refer to the appropriate section below for complete administration procedures. 



ON THE DAY OF THE TEST 



Admission t o the Testing Rorm 

About fifteen minutes before the test starts, you may begin admitting the examinees into 
the testing room. If an examinee arrives who did not register ahead of time, it is at your 
discretion whether to admit that examinee or not. Only those who have paid the $60 test 
fee should be allowed to take the test. 

rh^ckiny t;ifiiminee Identification 

All examinees must present proper identification (a photo ID such as a driver's license, 
student ID, or passport). The test supervisor must verily that the person who registered for 
the test is indeed the same individual who takes the test If an examinee forgets to bring 
any identification, he/she should not be admitted into the testing room. There can be no 
exceptions to this rule, even if a test supervisor knows the examinee veiy well 

Prohibited M.ti>ri.i€ 

While taking the test, examinees should have nothing on their desks but their test 
booklets. The use of books, dictionaries, personal recording devices, photographic devices, 
pens, notes, and papers of any kind is prohibited during the test 

Collecting Request Forms for Official Sco re Reports 

Each examinee should present you with a completed copy of the Request Form for 
Official Sgprg Reports. You should return these forms to CAL with the test materials. 
Note that a student may dedde to cancel the score reports after taking the exam. In this 
case, simply have the student completely cross i ut the applicable address(es) on the req jest 
form. 
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lANGUAGE lABORATORY ADMINISTRATION 



IMPORTANT NOTE: It is the responsibility of the institution to make sure that the 
examinee response tapes can be listened to on standard equipment Therefore, it is 
recommended that the PST be administered in a language laboratoiy with a quarter- 
track system. Such a system ensures that the tape containing student responses can 
be listened to on a standard portable cassette recorder. Some language laboratories 
use a two-track system in which student responses are recorded on the bottom half of 
the tape. Tapes recorded in this manner can not be listened to on a standard portable 
cassette player. If the institution has only a two-track system, arrangements may be 
made with CAL to have the tapes re-recorded. However, these arrangements would 
incur an additional time delay of up to two weeks in reporting the test results and an 
•dditioiial cost of $15.00 per examinee tape. 



At least 24 hours prior to the test, please make sure that the following are available 
and are in proper orden 

(1) A copy and a back-up of the master test tape for the particular version of the test 
being administered (Form A, Form B, or Form Q. 

(2) As many copies of the printed test booklet as there are students to be tested. Please 
make sure that the version shown on the front cover of the test booklet (Form A, 
Form B, or Form Q corresponds to the version of the master test tape. 

(3) For each student to be tested, a blank C:2D cassette tape, capable of 45 minutes of 
recording on qb^^. Shorter tapes (e.g., C60, with a 30-minute running time) will 
not work properly, since the test tuns continuously for almost 45 minutes. Each 
student cassette should have a label on which the student's name, test form and 
version, and institution are to be written. 

(4) Or.c or more fine-line felt tip markers for writing on the cassette tape label the name 
of the examinee, the institution where the testing takes place, the PST form 
administered (A, B, or C) and the PST venion (Brazilian or Lusitanian). 

On the day of the test, direct each student to a language laboratory booth in a random 
inanner. Do not allow students to self-select their seating positions. Wherever possible, 
allow at least one empty booth between students. When all students have been seated, 
distribute a ManK »»CttC tapg and one copy of the test booklet to each student. TeU the 
students not to open the test booklets untfl instructed to do so by the master test tape. 

Tell each student to write his/her Mjiamfe test form ktter (Form A, B or Q. version 
PI the tcrt (Brazilian or Lusitanian) and the name of the institution where the test is being 
administered on the cassette tape. After this is done, say the following: 
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We arc just about ready to administer the Portuguese Speaking Test, which will take 
appronmately 45 minutes. Now, please make sure that your cassette is properly 
inserted in the recording unit at your booth. Do nsa start the tape recorder at this 
time. If there are any problems, laise your hand. 



After any problems have been taken care of, say: 



The test questions and all directions for the test are recorded on the master test tape, 
which I will begin to play for you in a moment Once the test starts, it wm run through 
to the end. All you will need to do is start your tape recorder one time, when the 
voice on the master tape asks you to do so. After that, you should let the tape run 
through to the end, and foUow the various directions that will be given to you in the 
course of the test 



Note that in some language laboratories it is possible to start the students* machines 
automatically from the master console. In this case, instead of the above, say the following: 



The test questions and all directions for the test are recorded on the master test tape, 
which I will begin to play for you in a moment I will also start your tapes running 
from the master console, so you do not need to start your own machine when the 
master tape asks you to do so. I will go around the room and check to make sure that 
all machines are running properly. You need only to follow the various directions that 
will be given you in the course of the test 



Answer any questions concerning mechanical procedures, and then say: 



We are now ready to begin the test Please make sure that your headphones are on 
and the playback volume is adjusted so that you can bear the master tape clearly. I 
wfll now Stan the master tape. 



Start the master test tape. While the initial directions are being given by the master 
voice, walk around the room to make sure that the students* volume controls are properly 
adjusted and that the student cassettes are properly installed in the booth recorders. When 
the master voice says, "Please put your tape recorder in the 'reconT position and start the tape 
movmg through the machine," again check that all students have folk>wed these directions 
and that their machines are running and are in the "record* mode. 

The master tape will continue, automatically giving all further directions, test questions, 
and response pauses in the appropriate sequence. 
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At the end of the test, the master voice wOl say: "Stop. This is the end of the Portuguese 
Speaking Test. Please close your test booklet, stop your tape recorder, and wait for further 
instructions." At this point, make sure that all students have stopped recording (or stop their 
machines from the master console) and say: 



Please rewind your cassette, and also make sure that you have written your name, the 
letter of the test form, and the name of your institution on the cassette label. 



After all examinee tapes and the master tape are rewound, walk around the room, 
collecting both student tapes and test booklets. Make sun that vou have retrigvcd a tope 
•nd i krt bPOKIyl from web Itytfr nt When aU test booUets have been accounted for, the 
studenu may be dismissed. 

INDIVIDUAL ADMINISTRATION 

For each testing room, the following shouM be availab'e and in proper order: 

(1) Two portable cassette recorders, one to play the roaster test tape aloud and one to 
record the student's responses. If at all possible, the unit used to record the student 
responses shouM have "automatic volume contror in the recording mode. If this is 
not the case, attention wiU need to be paid to the proper adjustment of the recording 
level control, so that the student's voice will be clearly recorded at a suitable volume. 

(2) A copy of the master test tape for the particular version of the test being 
administered (Form A, Form B, or Form Q. 

(3) One or more copies of the printed test booklet. If students are to be tested in 
sequence, usmg the same test booUeL please examine the test booklet after each 
student has been tested to make sure that no notes or other markings have been 
made in the test booklet 

(^) A blank C-90 ai?cnc tape (one per student), with a label on which the student's 
name and institution are to be written. The C-90 length is needed to record the 
entire test <m a single nde. 

(5) A fine-line felt tip marker for writing on the cassette tape label the name of the 
examinee, the institution whfc:c the testing takes pla.^ the PST fonn administered 
(A, B, or C) and the PST version (Brazilian or Lusitanian). 

On the day of the test, set up the recording equipment and test materials in a quiet, 
comfortable room. The best arrangement is to use a table at least 3 feet in each dimension, 
on which both tape recorders and the test booklet are placed. The unit on which the 
student s responses wfll be recorded should be placed as close to the student as conveniently 
possible (within 18 inches) and the student should have room to spread out the test booklet 
comfortably on the toble surface. Hie cassette unit playmg back the master test tape may 
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be somewhat farther away, but near cnoush to the student s recorder for the latter to pick 
up the master voice as well as the student's own answers. 

Ask the student to write his/her Mjiamc test form letter (Form A, B or C) and the 
name of the in?tityt|pn at which the test is being administered on the cassette label. A 
felt-tip pen is preferable. You or the rtudent should then insert the blank cassette in the 
student's recorder. 



Then say: 



We are just about ready to administer the Portuguese Speaking Test, which will take 
approxiniately 45 minutes. One of the tape recorders [point to recorder] will play the 
master test tape, which will provide the test questions and all directions fcr the test. 
The other recorder [point to recorder] wfl] be used to record your responses. Once the 
test starts, it wiU run through to the end. All you wfll need :o do is start your own tape 
recorder [pointj one time, when the voice on the master tape asks you to do so. After 
that, you shouM let the tape run through to the end, and foMow the various directions 
that will be given you in the course of the test 



Answer any questions concerning mechanical procedures, and then say: 



I will now start the master tape. 



Start the master test tape. During the initial directions being given by the master voice, 
make sure that the playback volume for the master voice is at a clear and comfonable 
level and that the student's cassette is properly inserted in the second recorder, if the 
students recorder does m have automatic recording volume control, make sure that the 
recordmg level control is set high enough to record the student's voice clearly and without 
dutortion. ' 



When the master voice says, 'Please put your tape recorder in the 'iecortT position and 
Stan the tape moving through the machine." make sure that the student has foUowed these 
directions properly and that the student's recorder is running and is in the "Record* mode. 

The master tape wiU continue, aut jmatically ghing all further directions, test questions, 
and response pauses in the appropriate sequence. 

At the end of the test, the master voice will say: "Ao/?. This is the end of the Portuguese 
Speaking TaL Please dose your test booklet, stop your tape recorder, and wait for further 
utstmcaons. Rewind both the master test tape and the student response tape. Make sure 
that the student's name, test form letter and the name of the institution are on the student 
cassette label. Verify that the test booklet is in your possession. The student may then be 
dismissed. ' 
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AFTER THE TEST 



Checking Collected Material* 
Immediately after you dismiss the examinees, please perform the following checks. 

1. Check to see that master test tape has been completely rewound. 

2. Spot-check examinee cassettes to make sure that responses have been properly 
recorded. 

3. Flip through the test booklets to make sure that there are no pencil marks or missing 
pages. Report any defective or missing test booklets or test tapes on the Supervisor's 
Rcpon Form-^ Also be sure to include any other irregularities such as cheating (pve 
the examinee's name), malfunctioning equipment, mistiming, absence of examinees 
due to sickness, etc 

Refaniing Test Material tn C\l 

Within threedays after the test is given, please return the materials requested on the 
Tffl MPtCri?!? ChgfKli?^ to CAL Send them to the address given on page 5. Be sure to 
fill out and sign the Supervisor's Report Form and check off on the Test Materials Checklist 
all materials being returned. 
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INTERPRETATION OF TEST RESULTS 



The rating PST exam inees receive is based on the Generic Speaking Proficiency 
Guidehnes developed by ACTFL (American Council on the Teaching of Foreign Languages, 
1986). According to the introduction to the guidelines, they: 

represent a hierarchy of global characterizations of integrated performance in 
speaking, listening, reading and writing. Each description is a representative, not an 
exhaustive, sample of a particular range of ability, and each level subsumes all 
previous levels, moving from simple to complex in an "all-before-and-more" iashioa 

These guidelines focus on proficiency, ic^ what an individual can and cannot do in the 
language, rather than on achievement, Le^ what an individual has achieved through specific 
language courses. The guidelines, thus, are based on a global assessment of the examinee's 
current abib'ty to use the language. 

Below are the general descriptions for the relevant ACTFL profidency levels. The PST 
IS designed for students between Intermediate-Low and Superior on this scale. 

Novice 

The Novice level is characterized by the ability to communicate, minimally with memorized 
material. The PST is designed for aaminees who exceed this level. Any examinee no* 
achievmg the mininum ability to be rated at the Intermediate level received this rating. 



latcnncdiate 

The three Intermediate levels are characterized by the speaker's ability to: 

create with the language by combining and recombining learned elements, though 
primarily in a reactive mode; 

initiate, minimally sustain, and dose in a simple way basic communicative tasks; and 
* ask and answer questions. 



Intermediate-Low Able to handle successfully a limited number of interactive, task- 
oriented and social situations. Can ask and answer questions, initiate 
and respond to simple statements and maintain fece-to-face 
conversation, ahhough in a highly restricted ir anner and with much 
linguistic inaccuracy. Within these limitations, can perform such tasks 
as introducing self; ordering a meal, asking directions, and making 
purchases. Vocabulary is adequate to express only the most elementary 
needs. Strong interference from native language may occur. 
Misunderstandings frequently arise, but with repetition the 
Intermediate-Low speaker can generally be understood by sympathetic 
interlocutors. 

16 



1^ 



S3 



Intermediate-Mid Able to handle successfully a variety of uncomplicated, basic and 
communicative tasks and social situations. Can talk simply about self 
and family members. Can ask and answer questions and participate in 
simple conversations on topia beyond the most immediate needs; e.g., 
personal histoty and leisure time activities. Utterance length increases 
slightly, but speech may continue to be characterized by frequent long 
pauses, since the smooth incorporation of even basic conversational 
strategies is often hindered as the speaker struggles to create 
appropriate language forms. Pronunciation may continue to be strongly 
influenced by first language and fluency may still be strained. Although 
misunderstandings still arise, the Intermediate-Mid speaker can 
generally be understood by ^pathetic interlocutors. 

Intermediate-High Able to handle successfully most uncomplicated communicative tasks 
and social situations. Can initiate, sustain, and ckise a general 
conversation with a number of strategies appropriate to a range of 
circumstances and topics, but errors are evident. Limited vocabulaiy 
still necessitates hesitation and may bring about slightly unexpected 
circumlocution. There is emerging evidence of connected discourse, 
particularly for simple narration and/or description. The Intermediate- 
High speaker can generally be understood even by interlocutors not 
accustomed to dealing with speaken at this level, but repetition may 
still be required. 

Alvanced 

The two Advanced levels are characterized by the speaker's ability to: 

• converse in a clearly partidpatoiy fashion; 

• initiate, sustain, and bring to closure a wide variety of communicative tasks, including 
those that require an increased ability to convey meaning wth diverse language 
strategies due to a complication or an unforeseen turn of events; 

• satisfy the requirementt of school and work situations; and 

• narrate and describe with paragraph-length conneaed discourse. 

Advanced Able to satisfy the requirements of eveiyday situations and routine 

school and work requirements. Can handle with confidence but not 
with fadb'ty compUcated tasks and social situations, such as elaborating, 
complaining, and apologizing. Can narrate and describe with some 
details, linking sentences together smoothly. Can communicate facts 
and talk casually about topics of current public and personal interest, 
using general vocabulaiy. Shortcomings can often be smoothed over 
by communicative strategies, such as pause fillers, stalling devices, and 
different rates of speech. Circumlocution which arises from vocabulaiy 
or syntactic limitations veiy often is quite successful, though some 
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groping for words may still be evident. The Advanced-level speaker 
can be understood «rithout difficulty by native interlocutors. 



Advanced-Plus Able to satisfy the requirements of a broad variety of everyday, school, 
and work situations. Can discuss concrete topics relating to particular 
interests and special fields of competence. There is emerging evidence 
of ability to support opinions, explain in detail, and hypothesize. The 
Advanced-Plus speaker often shows a well developed ability to 
compensate for an imperfect grasp of some forms with confident use 
of communicative strategies, such as paraphrasing and circumlocution. 
Differentiated vocabul'iy and intonation are effectively used to 
communicate fine shades of meaning. The Advanced-Plus speaker 
often shows remarkable fluency and ease of speech but under the 
demands of Superior-level, complex tasks, language may break down 
or prove inadequate. 

Sopeiior 

The Superior level is characterized by the speaker's ability to: 

participate effectively in most formal and informal conversations on practical, sodal, 
professional, and abstract topics; and 

support opinions and hypothesize using native-like discourse strategies. 



Superior Able to speak the language with sufficient accuracy to partidpatc 

effectively in most formal and informal conversations on practical, 
social, professional, and abstract topics. Can discuss special fields of 
competence aiid interest with ease. Can support opim'ons and 
hypothesize, but may not be able to tailor language to audience or 
discuss in depth highly abstract or unfamiliar topics. Usualfy the 
Superior level speaker is only partially familiar with regional or other 
dialectical variants. The Superior level speaker commands a wide 
variety of interactive strategies and shows good awareness of discourse 
strategies. The latter involves the ability to distinguish main ideas from 
supporting information through syntactic, lexical and suprasegmental 
features (pitch, stress, intonation). Sporadic errors may occur, 
particulars^ in low-L-^uency structures and some complex high- 
frequency structures more coimnon to formal writing, but no patterns 
of error are evident. Errors do not disturb the native speaker or 
interfere ivith communication. 

High-Superior This rating, which is not part of the ACTFL scale, is used in PST 
scoring for examinees who clearly exceed the requiremenu for a rating 
of Superior. A rating of High Superior corresponds to a rating of 3+ 
to 5 on the scale used by the Interagency Language Roundtable (ILR) 
of the VS. government. The PST is not designed to discriminate 
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among examinees above the ACTFL Superior level. A rating of High- 
Superior means that the examinee's proficiency is equivalent to at least 
flj± on the ILR scale. 
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TECHNICAL INFORMATION ON THE PST 



Development of the PST 

During the past decade, thanks prima rily to the efforts of the American Cbuncil on the 
Teaching of Foreign Languages (ACTFL) and with the assistance of several government 
language training agencies under the auspices of the Interagency Language Roundtable 
(ILR), a major theoretical and practical development in the field of foreign language 
assessment has taken place. This development is the application of a criterion-referenced 
orientation to the testing of foreign language proficiency. 

Two standards lie at the heart of the oral proficiency testing movement: the Oral 
Proficiency Interview (OPI) and the ACTFL/ILR speaking proflciency guidelines. TheOPI 
is a direct face-to-face evaluation of the learner's second language competence conducted 
by trained interviewers and raters. [For information on the direct oral proficiency interview 
used by the government see Wilds (1975) and SoUenberger (1978). For information on the 
ACTFL oral interview see Higgs (1984), and Stansfield and Harmon (1987)]. The ACTFL 
proficiency guidelines, based on the earlier developed ILR guid elines, provide the criteria 
against which language proficiency is rated. Since 1980, ACTFL has trained approximately 
1,000 interviewers and raters in the major foreign languages taught in the United States: 
Spanish, French, German ana Russian. As a result, oral proficiency testing is widely 
available to individuals whose competence in these languages must be rated. However for 
less commonly taught languages such as Portuguese, the scarce number of trained 
interviewers makes it difficult if no> impractical to connect an interviewer in one part of the 
country with an examinee in another part of the country. This situation impedes the 
availability of an oral proficiency interview to an individual who may have need for such a 
rating. The Portuguese Speaking Test (PST) was developed in response to this need. 

The PST, a semi-direct test of oral proficiency, is firmly based on the above mentioned 
ACTFL/ILR standards. Its format seeks to emulate the OPI for intermediate and advanced 
level students by its various linguistic and so cio-lin guistic probes of examinee speaking 
ability. It is scored accor ding t o the current ACTFL/ILR oral proficiency guidelines, and 
examinees are given an ACTFL level description as their score (see Interpretation of Test 
ResulU, pp. 15-18). 

The PST was developed by the Foreign Language Education and Testing Division of the 
Center for Applied Linguistics (CAL) in 1988 through a grant from the Office of 
International Research and Studies of the U.S. Department of Education. The format of 
the test was based on an earlier semi-direct test of oral proficiency developed at CAI^ the 
Chinese Speaking Test (Clark and Li, 1986). First, three preliminary forms of the PST, each 
containing more items than would ultimately be required, were developed. These three 
forms were field-tested on two native speakers (one Brazilian and one Lusitanian) in order 
to ensure that the questions were clear and culturally appropriate. They were also field 
tested on 15 students of Portuguese at Georgetown University to ensure that the test 
questions were clear, understandable and working as intended, and to check the 
appropriateness of the pause times allotted for examinees' responses on the tape. Based on 
direct observation of examinees taking the test, written and verbal feedback, and evaluations 
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of the speech samples elidted by each test question, some minor revisions were made in the 
test directions, test questions, visual stimuli, and pause length. Following these revisions a 
formal study of the reliability and validity of the final three forms of the PST was carried 
out in May 1988. 

In this study, 30 students of Portuguese, at a broad range of abilities and of widely 
diverse backgrounds, were adm inister ed two different forms of the PST as well as :he OPI 
during a two<lay period. An ACTFL certified and an ILR certified rater were used. For 
a given subject, the OPI was administered and the subject rated by one of these two raters. 
Each OPI was recorded and scored at a later date by the other rater. Each rater also 
scored all the PST tapes. Thus, each OPI and each form of the PST was scored 
independently by each rater using the same ACTFI7ILR scale. The data collected from 
the study serves as the basis for the statistical information concerning reliability and validity 
presented in the following sections. The complete technical report of the study, which 
includes a description of the characteristics of the sample, raw data and more in-depth 
statistical analyses than can be presented here, is available on microfiche in Stansfield and 
Kenyon (1988). 

Level ofDifficnltv 

The PST is designed to be a measure of speaking proficiency. Obviously, speaking 
proficiency can vary according to the type of task that an examinee is presented. Experience 
gained in developing the PST indicates that the tasks it requires an examinee to perform 
invohre extended oral produ ction and can not be addressed by an examinee who is at the 
Novice level on the ACTFL scale or below level 1 on the ILR scale. This experience is 
similar to that of Clark and Swinton (1979) in the development of the Test of Spoken 
English and by Clark and Li (1986) in the development of the Chinese Speaking T.^st. The 
tasks employed in the PST are appropriate for intermediate and advanced speakers of the 
language. For the most advanced speakers (level 3+ to 5 on the ILR scale) the tasks may 
be appropriate, but the semi-direct format of the test does not permit the extensive probing 
of weaknesses necessary to differentiate among individuals at this level. The test is 
appropriate for examinees from Intermediate-Low to Superior on the ACTFL scale and 
from 1 to 3 on the ILR scale. 

Soeedednesi 

During the development of the PST, both native speakers and learners of Portuguese 
were observed while taking the test by CAL staff and members of the test development 
team. These careful observations were used to establish the length of the pauses allowed 
for the response to each question on the PST. As a result, the length of time allotted for 
the response to each question varies throughout the test. During the PST validatif ;n study, 
the 30 subjecu were asked on a questionnaire to rate the pause lengths associated with the 
PST. 70% of the subjects rated the pause lergth as "about right," while 19% rated the 
pauses as "too short" and 11% rated the pauses as "too long." Thus, it may be concluded 
that an adequate amount of time u allotted for most examinees to answer each question. 
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However, it should be noted that PST raters are instructed not to consider whether an 
examinee finishes his/her response before the '.:xt question begins. If an examinee does 
not finish, the rater merely evaluates the level of proficiency indicated by the speech that 
was elicited. During the validation of the PST, raters were also asked to indicate the 
adequacy of pause length for the purpose of making an evaluation of examinee speech. 
Raters indicated no interference in rating tapes due to pause length. 

Reliability of the PST 

Reliability is the extent to which a test yields consistent results. There are several 
different methods of measuring reliability that may be relevant to a single test. In a test 
such as the PST, where scores depend on the ability of trained raters to use the same 
scoring guidelines, the most important type of reliability is interrater reliability. This is the 
degree of consistency between different raters scoring the same examinee. For each of the 
PST forms, the interrater reliability (degree of correlation between two raters* score for any 
particular examinee) for the two raters who participated in the validation study is given 
below: 



PST Interrater Reliability 

Form A (N « 19) .93 
Form B (N « 21) .98 
Form C (N « 20) .96 

These interrater reliabilities are all unifonnly high across the three test forms. This 
suggests that the raters used to score the PST are highly consistent in the ratings they give 
to the PST speech samples. 

It is also important to know the degree to which each of the three forms of the PST 
are parallel, Le, how an examinee's score may be affected by which form of the test he/she 
took. The table below shows the consistency with which an examinee was rated on two 
different forms of the PST. 



Parallel Form ReUability 

^ RaiSLi Rater 2 Rater 1 x Rater 2 

Forms A an'* B (N « 10) .99 .97 .98 

Forms A and C (N « 9) .99 .93 94 

Forms B and C (N « 11) .99 .97 .97 

First Form/Second Form .99 .96 .95 

(N-30) 



In the table above, Tirst Form/Second Form" refers to the first and second form of the 
PST an examinee was administered. It is used here as a logical method to combine the three 
PSr forms. The figures in the first two columns show the parallel form reliabilities when 
the two different forms are rated by the same rater. The third column shows the 
reliabilities when Rater 1 scored the first form and Rater 2 scored the second form. The 
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statistics in this column thus show variation due not only to different forms, but also to 
different raters. 

These figures indicate that the parallel form reliability of the PST is veiy high even when 
different raters are used. The figures provide strong support for the conclusion that each 
form of the PST elicits a sample of speech that is uniformly challenging to the examinee and 
equally able to be evaluated consistently by the raters. The figures also warrant the 
conclusion that natural variations in examinee oral language performance are adequate^ 
controlled for by the PST format. 

Standard Error of M«««Mi»ingiit 

Any measurement of an individual's ability invoKes a degree of error, the magnitude of 
which depends on the reliability of the test This degree of error, when expressed in test 
score units, is called the standard error of measurement (SEKf). The SEM indicates the 
interval or range in test scores that would be expected if an examinee were to take parallel 
forms of the test on numerous occasions. Thus, the standard error provides a confidence 
interval within which we can safely claim that the examinee's score would stay if the 
examinee took the test again prior to aiiy real change in his or her proficiency. This is the 
practical usefiibiess of the SEM; it provides a kwer limit and an upper limit on the estimate 
of an oaminee's true ability. 

The standard error of measurement of a multiple-choice test can be calculated in a 
straightforward manner. However, when the examinee's score on a test involves human 
judgement, such as with the PST, the SEM depends in large part on the reUability of the 
indhddual rater. Therefore, it is not possible to state the SEM of the PST exartly. For the 
two raters used in the PST validity study, the SEM would be .11 of a level on the 0-5 ILR 
scale for one rater and .19 of an ILR level for the other. These SEM coefficients assume 
that all the examinee's test scores would be rated by the same rater. 

In the PST validity study, when the scores were rated by different raters, the standard 
error would be .23 of an ILR level. On the PST, this means that ratings at the lower level 
(Intermediate) are not as stable as those at the upper levels (Advanced and Superior). By 
applying this SEM (.23) to an examinee who obtained a rating of Intermediate Mid, we 
would find tfiat there is a 95% probability that the examinee would score either 
Intermediate Low, Intermediate Mid, or Intermediate High on a retest, if the retest were 
scored by a different rater (the most severe condition). There is a 5% probability that tiie 
examinee's score would fall ouUide this interval. A retest involving the same examinee and 
the same raters wouW be less likely to produce a change in rating at the higher PST levels. 
For example, an examinee who scored as Superior on the PST is more likely to obtain a 
score of Superior on a retest than any other score, and there is a 95% probability that his 
or her score wouW not be lower than Advanced Plus or higher than the original score of 
Superior. 

Thus, we can not precisely define the SEM of the PST, since it depends in large part on 
the rater or raters. Another reason it is difficult to define the SEM of the PST precisely is 
because the PST scale uses descriptive categories as opposed to numbers, which are typically 
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used for reporting scores on standardized tests. Nevertheless, it is appropriate to address 
the issue of the standard error, and we believe a reasonable estimate of the SEM is .20 of 
a level on the 0-5 ILR scale. While PST scores are quite accurate, this estimate of the SEM 
should be taken into account if veiy important or critical decisions are to be made based 
on a PST score. PST staff will be ^d to discuss the SEM with score useis in such cases. 

Validity of the PST 

According to the Standards for Educational a nd Pjevchnlnpcal T«>^t.n^ (American 
Psychological Association, 1985), test vaUdity refers to "the appropriateness, meaningfulness 
and usefulness of the specific inferences made from test scores" (p. 9). Validity is 
demonstrated by an accumulation of evidence that supports the claim of validity for a 
particular test Some of this evidence is empirical Other evidence may be qualitative, in 
that it deak with the content of the test, or ft may be theoretical, in that h deak with a 
theoiy iit>out the nature of the trait being measured by the test In the case of PST, the 
ceittral validity concern is the claim that the test is a measure of overaH speaking proficiency 
m Portuguese. 

Traditionally, three types of validity are usually identified according to how the 
evidence was gathered. These are content validity, criterion-related validity, and construct 
i^Iidity. The latter type, construct validity, focuses primarity on the test score as a measure 
of the psychotogical characteristic of interest" (American Psychological Association, p. 9). 
fai the case of the PST, this characteristic, or trait, is speaking proficiency in Portuguese. 
This IS a broad traft. composed of a complex of innate abilities and learned skflk. To date 
the most widely accepted measure of this trait is the OPI (see above). A great deal has 
been written elsewhere about the construct validity of the OPI, an J bo attempt is made to 
review this evidence here. [For information on the construct validity of the OPI see 
Bachman and Palmer (1981), dark (1978), Frith (1980). and Higgs (1984).) SufBce ft to say 
that followuig more than 30 years of use in the government and a decade of use ui 
academia, the construct validity of the OPI b regarded as high. Suice the PST was designed 
to be used as a surrogate OPI, ft would claim to measure essentially the same constructs 
as the OPI. and to the extent that the two tests foUow the same format and. roost 
importantly, produce the same results, there would be evidence for construct validity. 
Therefore, the following information on the content and criterion-related validity of the PST 
m relation to the OPI provides evidence of the constma validity of the PST as a measure 
of speaking proficiency that can be used when a direct uiterview is not feasible. 

Content validity is evidence that demonstrates the degree to which the sample of items, 
tasks or questions on a test are representative of the domain of content that could be tested 
jAjnencan Psychological Association, p. 10). Evidence for the content validity of the PST 
found in the tasks examinees are asked to perform ui demonstrating oral proficiency. 
First, students are asked to speak Portuguese only, and are not required to decode 
Portuguese (i.e., a diffeient task), since directions are given ui English. Ctonceming the 
&ce-to-face direct uiterview for which the PST may be used as a substitute, the speaking 
?*'^2J'^^"" ^^^^ *° perform on the PSl parallel those of the direct uiterview. Uke 
the OPI. the PST begins with a Varm-up" phase containing conversational questions about 
the examinee's personal background. It then moves to picture-based questions involving a 
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finite description. These questions function like the Tcvel check" phase of the OPI in that 
they allow the examinee to demonstrate his/her ability to describe or narrate in extended 
discourse. Then the PST moves to the text-based questions presented in English. This is 
analogous to the "probe" phase of the OPI. During this phase the examinee is instructed 
to talk about selected topics invohdng different discourse strategies. These strategies include 
explaining a process, describing a controversial topic and arguing in flavor of a particular 
point of view, and talking about a hypothetical situation. In the final part of the PST, the 
examinee is asked to imagine that he/she is confronting a particular problem in a particular 
situation and to use his/her persuasive powers to accomplish a desired end. For the OPI, 
and thus also the PST, these tasks have been deemed capable of probing an examinee's oral 
proficiency and of eliciting a representative sample of an examinee's speech. For further 
information relevant to the content validity of the PST, see the section on Test Content 
earlier in this manual 

In order to determine whether the PST is as effective as the OPI in probing an 
examinee's Ponuguese speaking proficiency, subjects in the validation study were asked to 
indicate on a questionnaire if the questions asked during each test (the OPI and the PST) 
adequately probed their maximum level of speaking ability. For both test formats the 
responses were identical: 73% said "Yes" whUe 27% said "No." These data provide further 
evidence of the content validity of the PST as a surrogate OPL 

However, unhlce the OPI, the PST is limited in the amount of probing it is able to do. 
Thus, for examinees at near-native and native levels of proficiency, such as levels 3+ 
through 5 on the ILR scale, the PST is not viewed as capable of probing as adequately or 
thoroughly as a direct interview. Since the PST is not viewed as an effective discriminator 
of individual differences in proficiency at these high levels, PST scores at these levels are 
combined into a single ratii^ of High-Superior. 

Criterion-related validity b evidence that "demonstrates that test scores are systematica^ 
related to one or more outcome criteria" (American Psychological Association, p. 11). In 
the case of the PST, the OPI is a relevant criterion variable and the relationship betjveen 
the PST and the OPI can be empirically measured. In the PST validation study (StansfieM 
and Kenyon, 1988), the following correlations were obtained the two raters who provided 
independent ratings of perfonnance on the OPI (Live Interview) and on two different forms 
of the PST. 

Correlations between PST and Live Interview (Same Rater) 



Form A (N« 19) 
Form B (N«21) 
Form C (N«20) 
All Forms (60 observations) 



In the table above as weU as the one following, "All Forms" refers to pairing each rater's 
60 mdependent PST scores (two PST forms s 30 subjects) with the corresponding OPI 



RalCLi 
.96 
.95 
.93 
.95 



Eil£L2 
.90 
.91 
.92 
.91 
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scores. The figures for this line, then, represent an average correlation. The high 
correlations presented above support the use of the PST as a measure of oral proficiency 
comparable to the OPl. The average correlation for the two raters is .93. The correlations 
supply empirical evidence for the Titerion-related validity of the PST as a measure of 
speaking proficiency in Portuguese. 

This criterion-related evidence is further supponed by the most demanding type of 
comparison: examinee perform£.-ce on the OPI (Live Interview) when rated by one rater 
with performance on the PST rated by a different rater. Below are the correlations between 
the OPI and the PST when scored by separate raters. 

Correlations between PST and Live Interview (Separate Raters) 

Form A (N« 19) .92 

Form B (N«21) .93 

Form C (N«20) .93 

All Forms (60 observations) .93 

In the PST validation study, each of the PST response tapes was rated by the two 
different raters and each interview was separately icored by the two different raters. The 
data in the table above indicate a high correlation between the PST and the OPI under the 
most stringent conditions; one rater rates the examinee on the PST while a different rater 
separately rates the examinee on the OPL 

A final summary correlation that pairs aU 60 independent interview ratings (30 
mtenriews, 2 raters) with the 120 independent PST ratings (60 ratings, 2 raters) b .93. This 
figure takes into account both same rater and different later scoring. Its magnitude, along 
with the other correlations above, strongly support the criterion-related and construct 
validity of the PST as a measure of oral proficiency and support its us* as an alternative to 
the OPI. 
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PORTUGUESE SPEAiQNG TEST 
Order Form 

Ship to: (Please print) Bill to: (if different from left) 



Name 


Name 




Position Title 


Position Title 




Department 


Department 




Institution 


Institution 




Street 


Street 




City/State/Zip Code 


City/State/Zip Code 




Telephone # 


Telephone # 





First order? 

Yes (If yes, CAL resenres the right to verify the above information) 

No 

Test Form. The Portuguese Speaking Test is available in three paraUel forms. 

Fonn A Form B Form C 

Tape VeraioD. The Portuguese Speaking Test master tape is available in two versions. 

Brazilian Portuguese Lusitanian Portuguese 

Number of ettmiBces expected: ($60 per examinee) 

Approxiiiute tesUiif date: Required deliveiy date: 

Purpose and u$t of the test: (Check all that apply) 

admission to a Portuguese study program 

placement within a Ponuguese study program 

exemption from Portuguese language requirement 

application for scholarship or appointment 

competency testing upon exit from a Portuguese program 

certification of Portuguese language proficiency for career purposes 

other (Please specify: . ' ) 



H7 



Payment 

You will be billed at $60.00 per examinee for the actual number of examinees tested 
after you return the test materiab to CAL Deliveiy charges will be paid bv CAL» provided 
this order form reaches CAL at least twow^ ks prior to the required deliveiy date indicated 
above. Dejiveiy charges for rush orders requiring a shorter delivery time will be billed to 
the institution placing the order. Official score reports will be issued about three weeks 
after the materials have been returned to CAL 

Sccurii. A{reemeiit 

It is vital that you maintain the security of the test materials from the time you receive 
them untii they are returned to CAL. Before the test materials are sent out, the following 
security agreement must be signed. 

/ promise to: 

(1) keep the PST test booklet and test tapes confidential and secure; 

(2) guarantee that neither the test booklet nor the test tapes will be reproduced in any form- 
and ' 

(3) rentm all test materials to the Center for Applied Linguistics within three days after 
administering the tesL 



Signature 


Date 


Name (print) 
Title 


Telephone nnmber 



Office Use Orly 

Date received: Date filled: By: 
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PORTUGUESE SPEAKING 

TEST 



EXAMINEE HANDBOOK 



All examinees mu.«t read this handbook before 
taking the Portuguese Speaking Test. 



IMPORTANT INFORMATION 



Examinee's Name 

Date of Exam 

Time of Exam 

Location ofExam 



Test Supervisor. 
Test Fee 



Institutional Registration? Yes 



No 



The Portuguese SpeaLLig Test was developed by the staff of the Division of Foreign Language Education and 
Testing at the Center for Applied Linguistics under Grant Number 0008740397 from the OfGce of International 
Research and Studies of the U.S. Department of Education. 



Copyrigjit (c) 1988 by Center for Applied Linguistics, Washington, DC All rights reserved 
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GENERAL INFORMATION 



The purpose of the Portuguese Speaking Test (PST) is to evaluate the level of oral 
proficiency in Portuguese attained by American and other English-speaking learners of 
Portuguese. It is Intended for post-secondary students and adults at intermediate and 
advanced levels of Portuguese. The test lasts about 43 minutes and consists of six parts. 
You will hear the directions and questions for all parts from a master test tape. The 
directions and test questions will be in English, wi h the exception of the test questions in 
Part One, which are in Portuguese. In addition, you will use pictures and text in a test 
booklet to answer some of the test questions. Your responses will be recorded on a separate 
cassette tape which will be evaluated by a specially trained PST rater. 

TEST REGISTRATION 

The PST must be administered by a qualified test supervisor. There are two ways to 
register: institutional registration and individual registration. If your instructor or institution 
has arranged for you to take the PST (called "institutional registration"), your institution will 
inform you of the date, time, and location of testing and of any testing fees to be charged 
to you. You may find that informatk)n written on the cover of this handbook. If not, you 
should record it there. Your institution may require a local registration for the exam. In 
this case, you will be required to present a photo identification card (passport, driver's 
license, or student ID containing a photograph of yourself) and give your name and social 
security number to the test supervisor before the day of the test. You may pay the testing 
fee (if applicable) to the test supervisor by cash or check when yon register or when you 
report to the test center on the day of the test If you opt to pay by check, confirm with 
your supervisor whether the check should be made payable to the supervisor or to the 
participating institution. Do not write a check to the Center for Applied Linguistics (CAL). 

K you are registering for the PST as an individual, please turn to Appendix A (page 9) 
for the INDIVIDUAL REGISTRATION PROCEDURES. 

PREPARING FOR THE TEST 

As a test of general oral proficiency, the PST is not based on any particular textbook or 
course of study, but on the kinds of language-use situations that you may encounter in every- 
day real-life contexts. Because this type of speaking proficiency can be achieved only through 
continued study and practice with the language, it will not be helpful for you to study 
intensively just prior to the test. The suggested preparation for the test is to carefully review 
the information contained in this manual, especially the descriptk)n of the test and sample 
questions, in order to know what to expect on the test. You may also want to practice 
speaking Portuguese as much as you can immeaiately prior to the test so that you will be 
Vanned up" at the time of the test. Additionally, since the PST requires you to sf>cak into 
a microphone to record your answers, you may wish to practice recording your own voice 
to help you get accustomed to this method. 



ON THE DAY OF THE TEST 



On the day of the test, you must report to the test center 10 minutes before the test is 
scheduled to begin; no one will be admitted after the test has begun. Remember to bring 
the following items with you: 

1. a photo ID (driver^s license, student ID, or passp ^ :); 

2. test fee (if applicab^f?) and you have not prepaid during registration; 

3. completed Request Form for Officia l Score Reports from the back of this handbook 
(see the section entitled OFFICIAL SCORE REPORTS below). 

If you forget to bring these items, you will not be admitted to the testing room. There can 
be no exceptions to this rule. 

The following materials are prohibited in the testing room: books, dictionaries, personal 
rtrording devices, photographic devices, pens, notes, and papers. Any cheating (giving or 
receiving assistance, taking the test for someone else, copying or recording any test material, 
etc.) will result in dismissal from the testing room and cancellation of your test results. The 
test fee will not be refunded under such circumstances. 

DESCRIPTION OF THE TEST 

The PST is designed to emulate the direct face-to-face Oral Proficiency Interview used by 
U> S. gov ernment agencies and the American Council on the Teaching of Foreign Languages 
(ACfhL) through a semi-direct format. Instea J of being asked questions in a face-to-face 
interview, you will hear directions and questions from a master test tape. For some of the 
questions you will also use a test booklet. Your responses v/ill be recorded on a second 
tape. After the test is concluded the answers you gave on the response tape will be 
evaluated by specially trained PST raters. Thus, your score will depend entirely on what 
is on your response tape. Therefore, it is necessary to always speak loudly and clearly into 
your microphone when taking the test. 

The PST may be administered to a group in a language laboratory setting or individually 
using two tape recorders. Your supervisor will determine the type of testing arrangements 
you will use. 

On the following page are the ge neral directions to the test which you will hear on the 
master test tape before the test begins. Read them now to acquaint yourself with them 
before you take the test. 

There are six parts of the PST. E:;ch requires you to accomplish different tasks with 
the language. The various types of questions are used to probe the depth of your ability to 
speak and use Portuguese. When the test is completed, your response tape will contain a 
representative sample of your present ability to speak Portuguese. The six parts of the 
PST are described below. 
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GENERAL DIRECTIONS 

This is a test of your ability to speak Portuguese fluently and accurately. There are 
several parts in the test. Directions for each part will be given in English. Listen 
carefully to the directions, and answer in Portuguese when you are asked to do so. 
Your answers will be recorded on tape, so it will be important for you to speak 
clearly enough and loudly enough for your voice to be properly recorded. 

During the test you will be asked to put yourself in various contexts in Brazil or 
Portugal. There is no need to accommodate your Portuguese to the geographic 
location of the different questions. Always use the type of Portuguese you normally 
speak. 

Within each part of the test, the speaking tasks you will be asked to perform will 
range from easy to considerably more challenging. Your are not expected to be able 
to answer all questions with equal facility. However, you should try to speak as much 
and as wel! as possible in response to each question. 

The amount of time provided for your response will vary depending on the 
3mplexity of the question. Since the amount of time eacu examinee needs to 
respond also varies, feel free to stop talking when you feel you have given a complete 
response. On the other hand, do not wony if you are occasionally interrupted by the 
next question before completing your answer. 

Remember that there are no right or wrong answers on this test. The purpose of this 
test is to allow you to demonstrate your ability to speak Portuguese. Therefore, 
always say as much as you can, speak as well as you can and show what you can do. 



Part L Personal Conversation. 

In this part, you will be asked to respond to several snort questions about your family, 
education, hobbies, etc. These short questions will be spoken in either Brazilian or 
Lusitanian Portuguese, depending on the master test tape requested by your supervisor. 
This is the only part in which Portuguese is used on the tape. After each question is asked, 
you will have between 5 and 25 seconds to respond, depending on the information requested 
by the question. Examples of these questions follow. 
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(You hear:) 




Voc6 vem de una famllia grande? 




O que i que pretende fazer logo ap6s o seu exame 


de portugu^? 


Voc£ acha que o estudo do portugu&s tern sido util 


e porquS? 



For each question in parts 2-6, you will be given between 15 and 30 seconds to prepare 
an answer before being required to speak. Time for giving an answer varies from 45 
seconds to 1 minute and 45 seconds, depending on the complexity of each question. 



Part 2. Giving Directions. 

You will be shown a pictorial map in the test booklet and be instructed to give directions 
between two points on the map. You will be told, in English, to whom to give the directions 
and why you need to give them. 

Part 3. Detailed Description. 

You will be shown a drawing depicting a variety of objects and actions in the test booklet 
and be instructed to describe the picture in as much detail as possible. You will be told, 
in English, to whom you are giving the description and why. 

Part 4. Picture Sequences. 

You will be instructed to speak in a narrative fashion about a sequence of four or five 
pictures shown on a single page in the test booklet. There are three picture-sequence 
questions focusing on present, past and future time narration respectively. Again, you will 
be told, in English, the context in which you are giving the narration. 

Part 5. Topical Discourse. 

In this part, you will be instructed to talk about five topics. You will hear the topics, in 
English, on the tape; they are also written in English in the test booklet. Examples of such 
topics follow. 
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Please describe the kinds of ]eisure*time activities you particularly enjoy. 

Talk about the advantages and disadvantages of using public transportation. 

Explain to a Portuguese*speaking friend how one would go about buying a used car in 
the United States. 

Some Americans feel that foreign language education should begin in the first grade 
or even kindergarten. Do you agree or disagree? Explain why. 

If you were financially able to spend one year traveling, free of any other 
responsibilities, where would you go and what would you do? 



Part 6. Situations. 

In this part, you will hear and read, in English, five printed descriptions of real-life situations 
in which a specified audience and communicative task are identified. You will then be 
instructed to cany out the specified task. Examples of such situations are given below. 



You are at a hotel in Brazil. Tell the clerk you need a double room with private bath 
for two nights. Ask about room rates and if you can pay by traveler*s check. 

You are at a clothing store in Lisbon. Ask if you can return a shirt you bought there 
yesterday. You discovered it had a stain on it when you brought it home. 

You want to buy a ceramic bowl at a stand ai an open air market in Lisbon. However, 
you feel the asking price is too high. Try to convince the shopkeeper to lower the 
price. 

When you arrive at the airport in Rio de Janeiro, you discover your luggage was sent 
to Europe. The clerk tells you it will be returned in three days. Irate, you ask the 
airiines to give you money for the new clothing you will need to buy for your important 
business meeting tomorrow. 

After staying with a large Brazilian family for the summer, you take them out to a 
formal restaurant to express your thanks. Before the meal begins, you formally give 
a little toast expressing your thanks for the family's hospitality during your stay. 
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Remember, there are no right or wrong answers on the test. Your score will depend on how 
you express yourself in Portuguese and not on the actual ideas you express. The entire test 
lasts about 43 minutes. You will speak Portuguese for about 20 minutes in total. 

INTERPRETING PST SCORE RESULTS 

Within three weeks of taking the PST, you will receive an official score report which will 
contain your oral proficiency rating. The rating you will receive is based on the ACTFL 
Proficiency Guidelines. These guidelines provide global characterization of speaking 
proficiency at different levels. Each level description is only representative of that range of 
ability and each higher level subsumes the characteristics of all levels beneath it. 

You must remember that these guidelines focus on profidency, i.e., what you can and 
cannot do in the language, rather than on achievement, i.e., what you have mastered in 
specific language courses. Thus, the guidelines are based on a global assessment of your 
current ability to use the language. 

On page 7 are the general descriptions for the ACTFL profidency levels used for the 
PST. This summary of level descriptions also appears on the back of your official score 
report. To interpret your PST score, locate the description for your rating. The description 
should be an adequate portrait of your present abili^ to speak Portuguese. 

OFHCIAL SCORE REPORTS 

After the test has been administered, the Center for Applied Linguistics (CAL) will arrange 
for your response tape to be rated by a certified PST rater. For each examinee, thice 
official PST score reports will be sent out within three weeks to the address(es) you request 
°n <hc Request Form for Offidal Score Reports. (This form is located on the back cover 
of this handbook and may be detached or photocopied.) You may have all three reports 
sent to yourself, or you may have CAL send one or two of your reports directly to an 
institution or employer. 

If you were registered through an institutional registration, a roster of scores will also 
be sent to the test supervisor at the institution where you were tested. If you did not submit 
a Request Form for Offidal Score Rennm, you may obtain your score from the test 
supervisor at that institution. (Note: If you registered individually, your scor^ reports will 
be sent to you but not to your test supervisor unless so requested on the Request Form for 
Official Score Repnm ;> 

You may also order additional offidal score reports at a cost of $6.00 each. They will 
be sent by CAL either to you or to an institution you designate. If you wish additional score 
reports, send a letter with your name, sodal security number, test date, test center and test 
supervisor (if known) along with the names and addresses of the places you want the score 
reports sent. Address the letter to: 

Coordinator, Portuguese Speaking Test 
Center for Applied Linguistics 
1118 22nd Street, NW 
Washington, DC 20037 
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NOVICE llic Novice level is characterized by the ability to commuiiicate minimally with 

learned material. The PST is designed for examinees who exceed this level. Any 
examinee not achieving the minimum ability to be rated at the btermediate level 
will receive this rating. 

INTERMEDIATE The Intermediate level is characterized by the speaker's ability lo: 

* create with the language by combining and reoombining learned elements, though 
primarily in a reactive mode; 

* initiate, minimallysusuin, and dose in a simple way basic communicative tasks; and 

* ask and answer questions. 



Intermediate-Low 



Intennediate-Mid 



Intermediate-High 



ADVANCED 



Advanced-Plus 



SUPERIOR 



High-Superior 



Able to handle successfully a limited number of interactive, lask-oriented and social 
situations. Misundersundings frequently arise, but with repetition, the Intermediate- 
Low speaker can generally be understood by sympathetic interlocutors. 

Able to handle successfully a variety of uncomplicated, basic and communicative 
tasks and soda! situatior s. Although misundersundings still arise, the Intermediate- 
Mid speaker can generally be understood by sympathetic interlocutors. 

Able to handle successfully most uncomplicated communicative tasks and social 
situations. The In ermcdiate-High speaker can generally be understood even by 
interlocutors not accustomed to dealing with speakers at this level, but rqietition 
may still be required. 

The Advanced level b characterized by the speaker's ability to: 
converse in a dearly partidpatory fashion - initiate, sustain, and bring to closure a 
wide variety of communicative tasks, including those that require an increased ability 
to convey meaning with diverse language strategies due lo a complication or an 
unforeseen turn of events; 

satisfy the requirements of school and work situations; and 
narrate and describe with paragraph-length connected discourse. 

In addition to demonstrating those skills characteristic of the Advanced level, the 
Advanced Plus level speaker b able to handle a broad variety of everyday, school, 
and work situations. There b emerging evidence of ability to support opinions, 
explain in deuil, and hypothesize. The Advanced-Plus speaker often shows 
remarkable fluency and ease of speech but under the demands of Superior-level, 
complex tasks, language may break down ov prove inadequate. 

The Superior level b characterized by the speaker's ability to: 

partidpate effectively and with ease in most formal and informal conversations on 

practical, social, professional, and abstract topics; and 

support opinions and hypothesize using native-like discourse strategies. 

Thb rating, which b not part of the ACTFL scale, b used in PST scoring for 
examinees who clearly exceed the requirements for a rating of Superior. A rating 
of High-Superior corresponds to a rating of to 5 on the scale used by the 
Interagency Language Roundtable of the U.S. Government. The PSTb not designed 
to evaluate examinees above the ACTFL Superior level. 
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Please include your payment, which may be made by check payable to PST/Cenier for 
Applied Linguistics, or by money order. Score reports will be sent within 2 weeks following 
receipt of your request by the PST office. 

RESCORING POLICY 

Once you receive your PST score, you may for any reason request that it be rescored. In 
this case, CAL will have it scored by a second PST rater. If this rating results in a change 
in your score CAL will send you a new score report and will report the new score to any 
institutions that were previously sent score reports by CAL. If there is no change in the 
score, you wiU be notified of this. The fee for rescoring a PST test is $15.00. If the 
rescoring results in a change in your score, this fee will be refunded to you. If you would 
like rescoring, send a check for $15.00 to the address above with a written request for 
rescoring. Be sure to include the following information: your full name, social security 
number, test date, test center location and test supervisor (if known). 

SCORE CANCELLATION REQUESTS 

You may cancel your test scores immediately after taking the PST if you send a written 
request to do so to the PST coordinator. This request must be received within seven Jays 
of caking the test. If you request a score cancellation, your test will not be evaluated and 
you will have no score. 

You may cancel your request to have an official score report sent to the institutions you 
designated on the Request Form for Official Score Reports by crossing out the address(es) 
on the form on the day of the test. Remember, however, that you must submit the request 
form if you want an official score report sent to you. 
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APPENDIX A 



INDIVIDUAL REGISTRATION PROCEDURES 

It is possible for an individual examinee who needs an oral proficiency score to initiate the 
testing procedure. In this case, you must follow the procedures described below. 

A Arrange for a qualified individual to supervise the test 

1. Cbntact the Center for Applied Linguistics (CAL) for the name of the PST test 
center nearest you. Call the Coordinator of the Portuguese Speaking Test at (202) 
429 9292 or write: b 6 v ; 

Coordinator, Portuguese Speaking Test 
Center for Applied Linguistics 
1118 22nd Street, NW 
Washington, DC 20037 

[NOTE; CAL is not able to accept collect phone calls.] 

2. CAL will inform you if a nearby test center is available. If so, then contact the test 
center supervisor directly. Fill out the Individual Repistration Form at the end of this 
section and give it to the test center supervisor. Instructions for completing this form 
are found below. Bring payment for the test ($60) to the test supervisor. Please note 
that individual test centers may charge an additional fee for the service of 
administering the test 

3. If a nearby test center is not available, follow these procedures. 

a. Contact an instructor at a local institution that offers a Portuguese language 
program. If no local institution offers Portuguese, contact the director of the 
languagr laboratory, or if none is available, a local foreign language teacher who 
is willing to administer the test. The PST coordinator may be able to offer you 
assistance in contacting an individual to act as test supervisor. However, it 
ren^ains your responsibility to get a test supervisor. 

b. Have the person who is willing to administer the test call or write CAL and 
request a PST Test Manual for complete instructions on administering the PST. 
As in Item 2 above, fill out the Individual Registration Form and give it to the 
test supervisor, who will forward it to CAL Bring payment for the test ($60) to 
the test supervisor. Please note again that individual test centers may charge an 
additional fee for the testing service. 

4. When registering with your test supervisor for the PST, you need to bring with you 
a photo identification card (driver's license, student ID, or passport) and give your 
name and social security number to the supervisor. 
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IMPORTANT: Under normal circumstances the PST will be administered about two weeks 
after CAL is contacted by a test supervisor. Be advised that finding a test supervisor may 
require considerable time if none is currently available. You are reminded to keep this in 
mind and plan accordingly. 

B. Complete the Individual Registration Form 

Individual Repstration Form is found at the end of this section. You may make a 
photocopy of it or detach it from this manual. It must be filled out by all examinees and 
their supervisors registering under the individual registration procedures described above. 
It consists of two parts. 

Part A: To be completed by the examinee. 

1. Please print your name and social security number clearly. Remember that your 
test supervisor will need to verify your social security number with a photo ID. 

2. Put the address where you want your results sent and a daytime and evening 
telephone number where you could be reached by the PST coordinator if 
necessary. 

3. Choose the PST version (Brazilian or Lusitanian Portuguese) that is right for you. 

4. Wr. your desired testing date and alternate testing dates. Remember that it 
takes two weeks to send the testing materials to your test supervisor once the PST 
office receives an order form from your supervisor. 

5. State your reason for taking the PST. This is optional, however, and is for CAL's 
informational use only. 

6. Do not forget to sign and date the form. 

Part B: To be completed by the test supervisor. 

1. Write the location where the test will be given. 

2. Print name, title, institution, address and telephone number of the supervisor. 

3. Check the examinee's photo ID and date and sign the form. 

This form is then sent along with the PST Order Form by the test supervisor to the PST 
office. 

Note: you may cancel your PST registration at any time. However, the amount of your 
refond will depend on when the registration is cancelled: 

Before the test materials are sent from CAL: $40.00 Refund 

After materials are sent but before the PST is administered: $30.00 Refund 
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PORTUGUESE SPEAKING TEST 



Individual Repstration Form 

(This fonn must be accompanied by an institutional PST Order Form available in the PST 
Tcrt Manyal or from CAL directly. Please follow all directions in the Examinee Handbook 
on completing this form .) 

Part A- To be filled out by examinee: 

Examinee Name: 

Last Fim MI 

Social Security Number: 
Address: 



Telephone Number: (day) 



PST Version: Brazilian Portuguese Lusitanian Portuguese 

Desired Testing Date: 

Alternate Testing Dates: ~ 
Reason for taking the PST (optional): 

/ verify thai the above information is correct. 



Date Signature of Examinee 

Part B: To be filled out by Test Supervisor. 
Test Center: 



Test Supervisor: 
Title of Supervisor: 
Institution: 
Address: 



Telephone: 

/ verify that I have agreed to serve as the test supervisor for the above examinee I will follow 
all directions and procedures stated in the PST Test Manual. Furthermore, I verify that the 
above examinee's identification and social security number are correct. 



Date Signature of Supervisor 
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PORTUGUESE SPEAKING TEST 

Request Form for Offficifll Score Reports 

Prini the required information below. Deuch this form and give it to the test center supervisor on the day of 
the test. ^ 



ExamineeName: 

Last 

Social Security Number: 



First 



Test Date: 



MI 



Test Center: 



(month / year) 



Test Supervisor (if known): 
Examinee's Address: 



Three copies of your official score rqwrt will be sent to the address above, unless you would like CAL to send 
the extra copies of your official score rqxirt directly to an institution or employer. If so, then print the namc(s) 
and address(es) below. To cancel this order, simply completely cross out the addressees) on the day of the exam 
after taking the PST. 



Name of score recipient 



Name of score recipient 



Institution 



Institution 



Department 



Department 



Number and Street 



Number and Street 



aty 



Gly 



Sute/Zip code 



Sute/Zip code 



Your signature 



Date 



Date received: 



Office Use Only 
Date processed: 



By: 
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OFnCIAL SCORE REPORT 



The examinee 



Portuguese Speaking Test and achieved a rating of 



has taken the 



Testing Site: 
Testing Date: 



Not official without this seal 



Center for 
Applied Linguistics 
1118 22nd Street. N.W. 
Washington. DC 20037 




NOVICE The Novice is characterized by the ability to communicate minimally with learned 

material. The PST is designed for examinees who exceed this level Any examinee 
not achieving the minimum ability to be rated at the Intermediate level will receive 
this rating. 

INTERMEDIATE The Intermediate level is characterized by the speaker's ability to: 

* create with the language by combining and recombining learned elements, though 
primarily in a reactive mode; 

* initiate, minimally sustain, and close in a simple way basic communicative tasks; and 

* ask and answer questions. 



Intermediate-Low 



Intermediate-Mid 



Intermediate-High 



ADVANCED 



Advanced-Plus 



SUPERIOR 



High-Superior 



Able to handle successfully a limited number of interactive, taskniriented and social 
situations. MisundersUndings frequently arise,but with rq>etition, the Intenncdiatc- 
Low speaker can generally be understood by symptbetic interlocutors. 

Able to handle successfully a variety of uncomplicated, basic and communicative 
tasks and social situations. Although misunderstandings still arise, the Intermediate- 
Mid speaker can generilly be understood by sympathetic interlocutors. 

Able to handle successfully most uncomplicated communicative tasks and social 
situations. The Intermediate-High speaker can generally be understood even by 
interiocutors not accustomed to dealing with speakers at this level, but rqietition 
may still be required. 

The Advanced level is characterized by the q)eaker*s ability to: 
converse in a clearly pnicipatory bshion - initiate, susuin, and bring to closure a 
wide vanety of communicative tasks, including those that require an increased ability 
to convey meaning with diverse language strategies due to a complication or an 
unforeseen turn of events; 

satisfy the requiremente of school and work situations; and 
narrate and describe with paragraph-length connected discourse. 

In addition to demonstrating those skills characteristic of the Advanced level, the 
Advanced Plus level speaker is able to handle a broad variety of cveiyday, school, 
and work situations. There is emerging evidence of ability to support opinions, 
explain in detail, and hypothesize. The Advanced-Plus speaker often shows 
remarkable fluency and ease of speech but under the demands of Superior-level, 
complex tasks, language may break down or prove inadequate. 

The Superior level is characterized by the speaker's ability to: 

prticipate effectively and with ease in most formal and informal conversations on 

practical, social, professional, and abstract topics; and 

support opinions and hypothesize using native-like discourse strategies. 

This rating, which is not part of the ACTFL scale, is used in PST scoring for 
examinees who clearly exceed the requirements for a rating of Superior. A rating 
of High-Superior corresponds to a rating of 3^ to 5 on the scale used by the 
Interagency Language Roundtable of the US. Government. The PST is not 
designed to evaluate examinees above the ACTFL Superior level 



ERLC 



